Vulnerability: Inefficient Regular Expression Complexity in nltk (word_tokenize, sent_tokenize) ### Impact
The vulnerability is present in [`PunktSentenceTokenizer`](https://www.nltk.org/api/nltk.tokenize.punkt.html#nltk.tokenize.punkt.PunktSentenceTokenizer), [`sent_tokenize`](https://www.nltk.org/api/nltk.tokenize.html#nltk.tokenize.sent_tokenize)  and [`word_tokenize`](https://www.nltk.org/api/nltk.tokenize.html#nltk.tokenize.word_tokenize). Any users of this class, or these two functions, are vulnerable to a Regular Expression Denial of Service (ReDoS) attack. 
In short, a specifically crafted long input to any of these vulnerable functions will cause them to take a significant amount of execution time. The effect of this vulnerability is noticeable with the following example:
```python
from nltk.tokenize import word_tokenize

n = 8
for length in [10**i for i in range(2, n)]:
    # Prepare a malicious input
    text = "a" * length
    start_t = time.time()
    # Call `word_tokenize` and naively measure the execution time
    word_tokenize(text)
    print(f"A length of {length:<{n}} takes {time.time() - start_t:.4f}s")
```
Which gave the following output during testing:
```python
A length of 100      takes 0.0060s
A length of 1000     takes 0.0060s
A length of 10000    takes 0.6320s
A length of 100000   takes 56.3322s
...
```
I canceled the execution of the program after running it for several hours.

If your program relies on any of the vulnerable functions for tokenizing unpredictable user input, then we would strongly recommend upgrading to a version of NLTK without the vulnerability, or applying the workaround described below.

### Patches
The problem has been patched in NLTK 3.6.6. After the fix, running the above program gives the following result:
```python
A length of 100      takes 0.0070s
A length of 1000     takes 0.0010s
A length of 10000    takes 0.0060s
A length of 100000   takes 0.0400s
A length of 1000000  takes 0.3520s
A length of 10000000 takes 3.4641s
```
This output shows a linear relationship in execution time versus input length, which is desirable for regular expressions.
We recommend updating to NLTK 3.6.6+ if possible.

### Workarounds
The execution time of the vulnerable functions is exponential to the length of a malicious input. With other words, the execution time can be bounded by limiting the maximum length of an input to any of the vulnerable functions. Our recommendation is to implement such a limit.

### References
* The issue showcasing the vulnerability: https://github.com/nltk/nltk/issues/2866
* The pull request containing considerably more information on the vulnerability, and the fix: https://github.com/nltk/nltk/pull/2869
* The commit containing the fix: 1405aad979c6b8080dbbc8e0858f89b2e3690341
* Information on CWE-1333: Inefficient Regular Expression Complexity: https://cwe.mitre.org/data/definitions/1333.html

### For more information
If you have any questions or comments about this advisory:
* Open an issue in [github.com/nltk/nltk](https://github.com/nltk/nltk)
* Email us at [nltk.team@gmail.com](mailto:nltk.team@gmail.com)
Vulnerability ID: GHSA-f8m6-h2c7-8h9x
CWE: CWE-400
Severity score: 7.5
Project: https://github.com/nltk/nltk
Published: 2022-01-06
File: nltk/tokenize/punkt.py
Commit message: Resolved serious ReDoS in PunktSentenceTokenizer (#2869)

* Resolved serious ReDOS in PunktSentenceTokenizer

* Improve performance by relying on string split instead of re.search

* Solved issue if sentence contains just one token
Commit author: Tom Aarsen

Vulnerable version (before fix):
 def word_tokenize(self, s):         return self._word_tokenizer_re().findall(s)      _period_context_fmt = r"""         \S*                          # some word material         %(SentEndChars)s             # a potential sentence ending         (?=(?P<after_tok>             %(NonWord)s              # either other punctuation def debug_decisions(self, text):         See format_debug_decision() to help make this output readable.         """          for match in self._lang_vars.period_context_re().finditer(text):             decision_text = match.group() + match.group("after_tok")             tokens = self._tokenize_words(decision_text)             tokens = list(self._annotate_first_pass(tokens))             while tokens and not tokens[0].tok.endswith(self._lang_vars.sent_end_chars): def sentences_from_text(self, text, realign_boundaries=True):         """         return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]      def _slices_from_text(self, text):         last_break = 0         for match in self._lang_vars.period_context_re().finditer(text):             context = match.group() + match.group("after_tok")             if self.text_contains_sentbreak(context):                 yield slice(last_break, match.end())                 if match.group("next_tok"):

Fixed version (after fix):
 def word_tokenize(self, s):         return self._word_tokenizer_re().findall(s)      _period_context_fmt = r"""         %(SentEndChars)s             # a potential sentence ending         (?=(?P<after_tok>             %(NonWord)s              # either other punctuation def debug_decisions(self, text):         See format_debug_decision() to help make this output readable.         """          for match, decision_text in self._match_potential_end_contexts(text):             tokens = self._tokenize_words(decision_text)             tokens = list(self._annotate_first_pass(tokens))             while tokens and not tokens[0].tok.endswith(self._lang_vars.sent_end_chars): def sentences_from_text(self, text, realign_boundaries=True):         """         return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]      def _match_potential_end_contexts(self, text):         """         Given a text, find the matches of potential sentence breaks,         alongside the contexts surrounding these sentence breaks.          Since the fix for the ReDOS discovered in issue #2866, we no longer match         the word before a potential end of sentence token. Instead, we use a separate         regex for this. As a consequence, `finditer`'s desire to find non-overlapping         matches no longer aids us in finding the single longest match.         Where previously, we could use::              >>> pst = PunktSentenceTokenizer()             >>> text = "Very bad acting!!! I promise."             >>> list(pst._lang_vars.period_context_re().finditer(text)) # doctest: +SKIP             [<re.Match object; span=(9, 18), match='acting!!!'>]          Now we have to find the word before (i.e. 'acting') separately, and `finditer`         returns::              >>> pst = PunktSentenceTokenizer()             >>> text = "Very bad acting!!! I promise."             >>> list(pst._lang_vars.period_context_re().finditer(text)) # doctest: +NORMALIZE_WHITESPACE             [<re.Match object; span=(15, 16), match='!'>,             <re.Match object; span=(16, 17), match='!'>,             <re.Match object; span=(17, 18), match='!'>]          So, we need to find the word before the match from right to left, and then manually remove         the overlaps. That is what this method does::              >>> pst = PunktSentenceTokenizer()             >>> text = "Very bad acting!!! I promise."             >>> pst._match_potential_end_contexts(text)             [(<re.Match object; span=(17, 18), match='!'>, 'acting!!! I')]          :param text: String of one or more sentences         :type text: str         :return: List of match-context tuples.         :rtype: List[Tuple[re.Match, str]]         """         before_words = {}         matches = []         for match in reversed(list(self._lang_vars.period_context_re().finditer(text))):             # Ignore matches that have already been captured by matches to the right of this match             if matches and match.end() > before_start:                 continue             # Find the word before the current match             split = text[: match.start()].rsplit(maxsplit=1)             before_start = len(split[0]) if len(split) == 2 else 0             before_words[match] = split[-1]             matches.append(match)          return [             (                 match,                 before_words[match] + match.group() + match.group("after_tok"),             )             for match in matches[::-1]         ]      def _slices_from_text(self, text):         last_break = 0         for match, context in self._match_potential_end_contexts(text):             if self.text_contains_sentbreak(context):                 yield slice(last_break, match.end())                 if match.group("next_tok"):

Patch:
patch: @@ -266,7 +266,6 @@ def word_tokenize(self, s):
         return self._word_tokenizer_re().findall(s)
 
     _period_context_fmt = r"""
-        \S*                          # some word material
         %(SentEndChars)s             # a potential sentence ending
         (?=(?P<after_tok>
             %(NonWord)s              # either other punctuation
@@ -1284,8 +1283,7 @@ def debug_decisions(self, text):
         See format_debug_decision() to help make this output readable.
         """
 
-        for match in self._lang_vars.period_context_re().finditer(text):
-            decision_text = match.group() + match.group("after_tok")
+        for match, decision_text in self._match_potential_end_contexts(text):
             tokens = self._tokenize_words(decision_text)
             tokens = list(self._annotate_first_pass(tokens))
             while tokens and not tokens[0].tok.endswith(self._lang_vars.sent_end_chars):
@@ -1333,10 +1331,68 @@ def sentences_from_text(self, text, realign_boundaries=True):
         """
         return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]
 
+    def _match_potential_end_contexts(self, text):
+        """
+        Given a text, find the matches of potential sentence breaks,
+        alongside the contexts surrounding these sentence breaks.
+
+        Since the fix for the ReDOS discovered in issue #2866, we no longer match
+        the word before a potential end of sentence token. Instead, we use a separate
+        regex for this. As a consequence, `finditer`'s desire to find non-overlapping
+        matches no longer aids us in finding the single longest match.
+        Where previously, we could use::
+
+            >>> pst = PunktSentenceTokenizer()
+            >>> text = "Very bad acting!!! I promise."
+            >>> list(pst._lang_vars.period_context_re().finditer(text)) # doctest: +SKIP
+            [<re.Match object; span=(9, 18), match='acting!!!'>]
+
+        Now we have to find the word before (i.e. 'acting') separately, and `finditer`
+        returns::
+
+            >>> pst = PunktSentenceTokenizer()
+            >>> text = "Very bad acting!!! I promise."
+            >>> list(pst._lang_vars.period_context_re().finditer(text)) # doctest: +NORMALIZE_WHITESPACE
+            [<re.Match object; span=(15, 16), match='!'>,
+            <re.Match object; span=(16, 17), match='!'>,
+            <re.Match object; span=(17, 18), match='!'>]
+
+        So, we need to find the word before the match from right to left, and then manually remove
+        the overlaps. That is what this method does::
+
+            >>> pst = PunktSentenceTokenizer()
+            >>> text = "Very bad acting!!! I promise."
+            >>> pst._match_potential_end_contexts(text)
+            [(<re.Match object; span=(17, 18), match='!'>, 'acting!!! I')]
+
+        :param text: String of one or more sentences
+        :type text: str
+        :return: List of match-context tuples.
+        :rtype: List[Tuple[re.Match, str]]
+        """
+        before_words = {}
+        matches = []
+        for match in reversed(list(self._lang_vars.period_context_re().finditer(text))):
+            # Ignore matches that have already been captured by matches to the right of this match
+            if matches and match.end() > before_start:
+                continue
+            # Find the word before the current match
+            split = text[: match.start()].rsplit(maxsplit=1)
+            before_start = len(split[0]) if len(split) == 2 else 0
+            before_words[match] = split[-1]
+            matches.append(match)
+
+        return [
+            (
+                match,
+                before_words[match] + match.group() + match.group("after_tok"),
+            )
+            for match in matches[::-1]
+        ]
+
     def _slices_from_text(self, text):
         last_break = 0
-        for match in self._lang_vars.period_context_re().finditer(text):
-            context = match.group() + match.group("after_tok")
+        for match, context in self._match_potential_end_contexts(text):
             if self.text_contains_sentbreak(context):
                 yield slice(last_break, match.end())
                 if match.group("next_tok"):
path: nltk/tokenize/punkt.py