CVE-2021-3842 (published 2022-01-04T15:15Z)

nltk is vulnerable to Inefficient Regular Expression Complexity

Severity: MEDIUM
CVSS v2 base score: 5.0 (AV:N/AC:L/Au:N/C:N/I:N/A:P)
CVSS v3 base score: 7.5, HIGH (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H)

Weakness information:
cwe_id: CWE-1333
cwe_name: Inefficient Regular Expression Complexity
description: The product uses a regular expression with an inefficient, possibly exponential worst-case computational complexity that consumes excessive CPU cycles.
extended_description: Some regular expression engines have a feature called "backtracking". If the token cannot match, the engine "backtracks" to a position that may result in a different token that can match.
                Backtracking becomes a weakness if all of these conditions are met:
                
                    The number of possible backtracking attempts are exponential relative to the length of the input.
                    The input can fail to match the regular expression.
                    The input can be long enough.
                
		
		  Attackers can create crafted inputs that
		  intentionally cause the regular expression to use
		  excessive backtracking in a way that causes the CPU
		  consumption to spike.
		
            
            

Fix information:
repo_url: https://github.com/nltk/nltk
commit_details: repo_url: https://github.com/nltk/nltk
author: Tom Aarsen
author_date: 2021-12-08 15:19:56+01:00
msg: Resolve ReDoS opportunity by fixing incorrectly specified regex (#2906)
repository: repo_url: https://github.com/nltk/nltk
repo_name: nltk/nltk
description: NLTK Source
repo_language: Python
owner: nltk
file_changes: filename: malt.py
change_type: ModificationType.MODIFY
diff: @@ -32,7 +32,7 @@ def malt_regex_tagger():
             (r"\)$", ")"),  # round brackets
             (r"\[$", "["),
             (r"\]$", "]"),  # square brackets
-            (r"^-?[0-9]+(.[0-9]+)?$", "CD"),  # cardinal numbers
+            (r"^-?[0-9]+(\.[0-9]+)?$", "CD"),  # cardinal numbers
             (r"(The|the|A|a|An|an)$", "DT"),  # articles
             (r"(He|he|She|she|It|it|I|me|Me|You|you)$", "PRP"),  # pronouns
             (r"(His|his|Her|her|Its|its)$", "PRP$"),  # possessive

programming_language: Python
filename: glue.py
change_type: ModificationType.MODIFY
diff: @@ -703,7 +703,7 @@ class Glue:
 
         regexp_tagger = RegexpTagger(
             [
-                (r"^-?[0-9]+(.[0-9]+)?$", "CD"),  # cardinal numbers
+                (r"^-?[0-9]+(\.[0-9]+)?$", "CD"),  # cardinal numbers
                 (r"(The|the|A|a|An|an)$", "AT"),  # articles
                 (r".*able$", "JJ"),  # adjectives
                 (r".*ness$", "NN"),  # nouns formed from adjectives

programming_language: Python
filename: brill.py
change_type: ModificationType.MODIFY
diff: @@ -329,7 +329,7 @@ class BrillTagger(TaggerI):
             )
             print(
                 "TRAIN ({tokencount:7d} tokens) initial {initialerrors:5d} {initialacc:.4f} "
-                "final: {finalerrors:5d} {finalacc:.4f} ".format(**train_stats)
+                "final: {finalerrors:5d} {finalacc:.4f}".format(**train_stats)
             )
             head = "#ID | Score (train) |  #Rules     | Template"
             print(head, "\n", "-" * len(head), sep="")

programming_language: Python
filename: brill_trainer.py
change_type: ModificationType.MODIFY
diff: @@ -91,7 +91,7 @@ class BrillTaggerTrainer:
     # Training
 
     def train(self, train_sents, max_rules=200, min_score=2, min_acc=None):
-        """
+        r"""
         Trains the Brill tagger on the corpus *train_sents*,
         producing at most *max_rules* transformations, each of which
         reduces the net number of errors in the corpus by at least
@@ -111,7 +111,7 @@ class BrillTaggerTrainer:
         >>> testing_data = [untag(s) for s in gold_data]
 
         >>> backoff = RegexpTagger([
-        ... (r'^-?[0-9]+(.[0-9]+)?$', 'CD'),   # cardinal numbers
+        ... (r'^-?[0-9]+(\.[0-9]+)?$', 'CD'),  # cardinal numbers
         ... (r'(The|the|A|a|An|an)$', 'AT'),   # articles
         ... (r'.*able$', 'JJ'),                # adjectives
         ... (r'.*ness$', 'NN'),                # nouns formed from adjectives
@@ -125,7 +125,7 @@ class BrillTaggerTrainer:
         >>> baseline = backoff #see NOTE1
 
         >>> baseline.evaluate(gold_data) #doctest: +ELLIPSIS
-        0.2450142...
+        0.2433862...
 
         >>> # Set up templates
         >>> Template._cleartemplates() #clear any templates created in earlier tests
@@ -137,7 +137,7 @@ class BrillTaggerTrainer:
         >>> tagger1 = tt.train(training_data, max_rules=10)
         TBL train (fast) (seqs: 100; tokens: 2417; tpls: 2; min score: 2; min acc: None)
         Finding initial useful rules...
-            Found 845 useful rules.
+            Found 847 useful rules.
         <BLANKLINE>
                    B      |
            S   F   r   O  |        Score = Fixed - Broken
@@ -150,7 +150,7 @@ class BrillTaggerTrainer:
           85  85   0   0  | NN->, if Pos:NN@[-1] & Word:,@[0]
           69  69   0   0  | NN->. if Pos:NN@[-1] & Word:.@[0]
           51  51   0   0  | NN->IN if Pos:NN@[-1] & Word:of@[0]
-          47  63  16 161  | NN->IN if Pos:NNS@[-1]
+          47  63  16 162  | NN->IN if Pos:NNS@[-1]
           33  33   0   0  | NN->TO if Pos:NN@[-1] & Word:to@[0]
           26  26   0   0  | IN->. if Pos:NNS@[-1] & Word:.@[0]
           24  24   0   0  | IN->, if Pos:NNS@[-1] & Word:,@[0]
@@ -162,11 +162,11 @@ class BrillTaggerTrainer:
 
         >>> train_stats = tagger1.train_stats()
         >>> [train_stats[stat] for stat in ['initialerrors', 'finalerrors', 'rulescores']]
-        [1775, 1269, [132, 85, 69, 51, 47, 33, 26, 24, 22, 17]]
+        [1776, 1270, [132, 85, 69, 51, 47, 33, 26, 24, 22, 17]]
 
         >>> tagger1.print_template_statistics(printunused=False)
         TEMPLATE STATISTICS (TRAIN)  2 templates, 10 rules)
-        TRAIN (   2417 tokens) initial  1775 0.2656 final:  1269 0.4750
+        TRAIN (   2417 tokens) initial  1776 0.2652 final:  1270 0.4746
         #ID | Score (train) |  #Rules     | Template
         --------------------------------------------
         001 |   305   0.603 |   7   0.700 | Template(Pos([-1]),Word([0]))
@@ -175,7 +175,7 @@ class BrillTaggerTrainer:
         <BLANKLINE>
 
         >>> tagger1.evaluate(gold_data) # doctest: +ELLIPSIS
-        0.43996...
+        0.43833...
 
         >>> tagged, test_stats = tagger1.batch_tag_incremental(testing_data, gold_data)
 
@@ -185,13 +185,13 @@ class BrillTaggerTrainer:
         True
 
         >>> [test_stats[stat] for stat in ['initialerrors', 'finalerrors', 'rulescores']]
-        [1855, 1376, [100, 85, 67, 58, 27, 36, 27, 16, 31, 32]]
+        [1859, 1380, [100, 85, 67, 58, 27, 36, 27, 16, 31, 32]]
 
         >>> # A high-accuracy tagger
         >>> tagger2 = tt.train(training_data, max_rules=10, min_acc=0.99)
         TBL train (fast) (seqs: 100; tokens: 2417; tpls: 2; min score: 2; min acc: 0.99)
         Finding initial useful rules...
-            Found 845 useful rules.
+            Found 847 useful rules.
         <BLANKLINE>
                    B      |
            S   F   r   O  |        Score = Fixed - Broken
@@ -212,7 +212,7 @@ class BrillTaggerTrainer:
           18  18   0   0  | NN->CC if Pos:NN@[-1] & Word:and@[0]
 
         >>> tagger2.evaluate(gold_data)  # doctest: +ELLIPSIS
-        0.44159544...
+        0.43996743...
         >>> tagger2.rules()[2:4]
         (Rule('001', 'NN', '.', [(Pos([-1]),'NN'), (Word([0]),'.')]), Rule('001', 'NN', 'IN', [(Pos([-1]),'NN'), (Word([0]),'of')]))
 

programming_language: Python
filename: sequential.py
change_type: ModificationType.MODIFY
diff: @@ -337,7 +337,7 @@ class UnigramTagger(NgramTagger):
         >>> test_sent = brown.sents(categories='news')[0]
         >>> unigram_tagger = UnigramTagger(brown.tagged_sents(categories='news')[:500])
         >>> for tok, tag in unigram_tagger.tag(test_sent):
-        ...     print("({}, {}), ".format(tok, tag))
+        ...     print("({}, {}), ".format(tok, tag)) # doctest: +NORMALIZE_WHITESPACE
         (The, AT), (Fulton, NP-TL), (County, NN-TL), (Grand, JJ-TL),
         (Jury, NN-TL), (said, VBD), (Friday, NR), (an, AT),
         (investigation, NN), (of, IN), (Atlanta's, NP$), (recent, JJ),
@@ -491,7 +491,7 @@ class AffixTagger(ContextTagger):
 
 @jsontags.register_tag
 class RegexpTagger(SequentialBackoffTagger):
-    """
+    r"""
     Regular Expression Tagger
 
     The RegexpTagger assigns tags to tokens by comparing their
@@ -503,7 +503,7 @@ class RegexpTagger(SequentialBackoffTagger):
         >>> from nltk.tag import RegexpTagger
         >>> test_sent = brown.sents(categories='news')[0]
         >>> regexp_tagger = RegexpTagger(
-        ...     [(r'^-?[0-9]+(.[0-9]+)?$', 'CD'),   # cardinal numbers
+        ...     [(r'^-?[0-9]+(\.[0-9]+)?$', 'CD'),  # cardinal numbers
         ...      (r'(The|the|A|a|An|an)$', 'AT'),   # articles
         ...      (r'.*able$', 'JJ'),                # adjectives
         ...      (r'.*ness$', 'NN'),                # nouns formed from adjectives
@@ -515,7 +515,7 @@ class RegexpTagger(SequentialBackoffTagger):
         ... ])
         >>> regexp_tagger
         <Regexp Tagger: size=9>
-        >>> regexp_tagger.tag(test_sent)
+        >>> regexp_tagger.tag(test_sent) # doctest: +NORMALIZE_WHITESPACE
         [('The', 'AT'), ('Fulton', 'NN'), ('County', 'NN'), ('Grand', 'NN'), ('Jury', 'NN'),
         ('said', 'NN'), ('Friday', 'NN'), ('an', 'AT'), ('investigation', 'NN'), ('of', 'NN'),
         ("Atlanta's", 'NNS'), ('recent', 'NN'), ('primary', 'NN'), ('election', 'NN'),

programming_language: Python
filename: demo.py
change_type: ModificationType.MODIFY
diff: @@ -393,11 +393,11 @@ def _demo_plot(learning_curve_output, teststats, trainstats=None, take=None):
     plt.savefig(learning_curve_output)
 
 
-NN_CD_TAGGER = RegexpTagger([(r"^-?[0-9]+(.[0-9]+)?$", "CD"), (r".*", "NN")])
+NN_CD_TAGGER = RegexpTagger([(r"^-?[0-9]+(\.[0-9]+)?$", "CD"), (r".*", "NN")])
 
 REGEXP_TAGGER = RegexpTagger(
     [
-        (r"^-?[0-9]+(.[0-9]+)?$", "CD"),  # cardinal numbers
+        (r"^-?[0-9]+(\.[0-9]+)?$", "CD"),  # cardinal numbers
         (r"(The|the|A|a|An|an)$", "AT"),  # articles
         (r".*able$", "JJ"),  # adjectives
         (r".*ness$", "NN"),  # nouns formed from adjectives

programming_language: Python