A Visual Guide to Regular Expression
Tags: python, nlp, regularexpression, regex
Published: 2020-10-23 10:34:52

It's a common task in NLP to either check a text against a pattern or extract parts from the text that matches a certain pattern. A regular expression or "regex" is a powerful tool to achieve this.  

While powerful, regex can feel daunting as it comes with a lot of features and sub-parts that you need to remember.

In this post, I will illustrate the various concepts underlying regex. The goal is to help you build a good mental model of how a regex pattern works.  

## Mental Model    
Let's start with a simple example where we are trying to find the word 'cool' in the text.

![](https://amitness.com/images/regex-mental-model-example.png)  

With regex, we could simply type out the word 'cool' as the pattern and it will match the word.  
```python
'cool'
```

While regex matched our desired word '**cool**', the way it operates is not at the word level but the character level. This is the key idea.
> **Key Idea**: Regex works at the character-level, not word-level.  

![](https://amitness.com/images/regex-working.png)  

The implication of this is that the regex `r'cool'` would match the following sentences as well.  

![](https://amitness.com/images/regex-exact-word-match.png)  

## Basic Building Blocks    
Now that we understand the key idea, let's understand how we can match simple characters using regex.

### a. Specific character  
We can simply specify the character in the regular expression and it will match all instances in the text. 

For example, a regular expression given below will match all instances of 'a' in the text. You can use any of the small and capital alphabets.  
```
'a'
```

![](https://amitness.com/images/regex-match-only-a.png)  

You can also use any digits from 0 to 9 and it will work as well.  
```
'3'
```

![](https://amitness.com/images/regex-python-3.7-example.png)  

Note that regex is case-sensitive by default and thus the following regex won't match anything.


```
'A'
```

![](https://amitness.com/images/regex-not-matched-by-capital-a.png)  

### b. White space character  
We can detect special characters such as whitespace and newlines using special escape sequences.

![](https://amitness.com/images/regex-white-space-characters.png)  

Besides the common ones above, we have:
- **\r** for carriage return
- **\f** for form feed
- **\e** for escape.

### c. Special sequences 
Regex provides a bunch of built-in special symbols that can match a group of characters at once. These begin with backslash `\`.

#### Pattern: `\d`
It matches any single-digit number between 0 to 9.  

![](https://amitness.com/images/regex-single-digit.png)  

Notice that matches are single digit. So we have 4 different matches below instead of a single number `18.04`.  

![](https://amitness.com/images/regex-ubuntu-18.04.png)  

#### Pattern: \s
It matches any whitespace character (<span style="color: #66bb6a;">space</span>, <span style="color: #f48fb1;">tab</span> or <span style="color: #7986cb;">newline</span>).  

![](https://amitness.com/images/regex-match-any-whitespace.png)  

#### Pattern: \w   
It matches any of the small alphabets(a to z), capital alphabets(A to Z), digits (0 to 9), and underscore.  

![](https://amitness.com/images/regex-slash-w.png)  

#### Pattern: .
It matches any character except the new line (\n).
  
![](https://amitness.com/images/regex-everything-except-newline.png)  

```python
import re

>>> re.findall(r'.', 'line 1\nline2')
['l', 'i', 'n', 'e', ' ', '1', 'l', 'i', 'n', 'e', '2']
```

#### Pattern: Negations  
If you use the capitalized versions of the patterns above, they act as negation. 

For example, if "\d" matched any digits from 0 to 9, then "\D" will match anything except "0 to 9".

![](https://amitness.com/images/regex-negation.png)  

### d. Character sets   
These are patterns starting with `[` and ending with `]` and specify the characters that should be matched enclosed by brackets.  

For example, the following pattern matches any of the characters 'a', 'e', 'i', 'o', and 'u'.
![](https://amitness.com/images/regex-aeiou.png)  

You can also replicate the functionality of `\d` using the below pattern. It will match any digits between 0 to 9.
![](https://amitness.com/images/regex-1-to-9.png)  

Instead of specifying all the digits, we can use `-` to specify only start and end digits. So, instead of `[0123456789]`, we can do:

![](https://amitness.com/images/regex-refactor-all-digits.png)  

For example, `[2-4]` can be used to match any digits between 2 to 4 i.e. (2 or 3 or 4).

![](https://amitness.com/images/regex-year-2014-example.png)  

You can even use the special characters we learned previously inside the brackets. For example, you can match any digit from 0 to 9 or whitespace as:  

![](https://amitness.com/images/regex-whitespace-or-digit.png)  

Below, I have listed some useful common patterns and what they mean.  

![](https://amitness.com/images/regex-common-pattern-for-bracket.png)  

### e. Anchors  
Regex also has special handlers to make the pattern only match if it's at the start or end of the string.  

We can use the `^` anchor to match patterns only at the start of a line. For example: 

![](https://amitness.com/images/regex-start-anchor.png)  

Similarly, we can use the `$` anchor after the character to match patterns only if it's the end of the line. For example:

![](https://amitness.com/images/regex-anchor-end.png)  

### f. Escaping metacharacters  
Consider a case where we want to exactly match the word "Mr. Stark". 

If we write a regex like `Mr. Stark`, then it will have an unintended effect. Since we know dot has a special meaning in a regex.

![](https://amitness.com/images/regex-dot-issue.png)  

So, we should always escape the special metacharacters like `.`, `$` etc. if our goal is to match the exact character itself.  

![](https://amitness.com/images/regex-dot-fixed.png)  

Here is the list of metacharacters that you should remember to escape if you're using them directly.  
```
^ $ . * + ? { } [ ] \ | ( )
```


## Repetition of basic blocks  
Now that we can pattern match any characters, we could repeat things and start building more complicated patterns.  

### a. Naive repetition
Using only what we have learned so far, a naive way would be to just repeat the pattern. For example, we can match two-digit numbers by just repeating the character-level pattern.  
```
\d\d
```

![](https://amitness.com/images/regex-slash-d-slash-d.png)  

### b. Quantifiers
Regex provides special quantifiers to specify different types of repetition for the character preceding it.

#### i. Fixed repetition
We can use the `{...}` quantifier to specify the number of times a pattern should repeat.

![](https://amitness.com/images/regex-manual-counts.png)

For example, the previous pattern for matching 2-digit number can be recreated as:

![](https://amitness.com/images/regex-it-is-2020.png)  

You can also specify a range of repetitions using the same quantifier. For example, to match from 2-digit to 4-digit numbers, we could use the pattern:

![](https://amitness.com/images/regex-min-max-count.png)  

When applied to a sentence, it will match both 4-digit and 2-digit numbers.  

![](https://amitness.com/images/regex-20-years-old.png)  

<div class="notice--warning">
<strong>Note:</strong>
<p>
There should not be any space between minimum and maximum count For example, \d{2, 4} doesn't work.
</p>
</div>

#### ii. Flexible quantifiers
Regex also provides quantifiers "*", "+" and "?" using which you can specify flexible repetition of a character.

- **0 or 1 times**: `?`  
The `?` quantifier matches the previous character if it repeats 0 or 1 times. This can be useful to make certain parts optional.    

    ![](https://amitness.com/images/regex-question-mark-clarify.png)  

    For example, let's say we want to match both the word "sound" and "sound" where "s" is optional. Then, we can use the `?` quantifier that matches if a character repeats 0 or 1 times.  
    ![](https://amitness.com/images/regex-question-mark-example.png)  

- **one or more times**: `+`  
The `+` quantifier matches the previous character if it repeats 1 or more times.

    For example, we could find numbers of any arbitrary length using the regex `\d+`.
    
    ![](https://amitness.com/images/regex-example-of-plus.png)  

- **zero or more times**: `*`  
The `*` quantifier matches the previous character if it repeats zero or more times.

## Usage in Python  
Python provides a module called "re" in the standard library to work with regular expression.

### Need for raw strings  
To specify a regular expression in Python, we precede it with **r** to create raw strings.
```python
pattern = r'\d'
```

To understand why we precede with **r**, let's try printing the expression **\t** without `**r**`.
```python
>>> pattern = '\t'
>>> print(pattern)

```
You can see how when we don't use raw string, the string `\t` is treated as the escape character for tab by Python.  

Now let's convert it into raw string. We get back whatever we specified.
```python
>>> pattern = r'\t'
>>> print(pattern)
\t
```

### Using re module  
To use `re` module, we can start by importing the `re` module as:
```python
import re
```

#### 1. re.findall
This function allows us to get all the matches as a list of strings.
```python
import re
re.findall(r'\d', '123456')
```
```
['1', '2', '3', '4', '5', '6']
```

#### 2. re.match
This function searches for a pattern at the beginning of the string and returns the first occurrence as a match object. If the pattern is not found, it returns None.    

```python
import re

match = re.match(r'batman', 'batman is cool')
print(match)
```

```
<re.Match object; span=(0, 6), match='batman'>
```
  
![](https://amitness.com/images/regex-match-object.png)  

With the match object, we can get the matched text as

```python
print(match.group())
```

```
batman
```

In a case where our pattern is not at the start of the sentence, we will not get any match.  
```python
import re

match = re.match(r'batman', 'The batman is cool')
print(match)
```

```
None
```

#### 3. re.search
This function also finds the first occurrence of a pattern but the pattern can occur anywhere in the text. If the pattern is not found, it returns None.    

```python
import re

match = re.search(r'batman', 'the batman is cool')
print(match.group())
```

```
batman
```

## References
- A.M. Kuchling, ["Regular Expression HOWTO - Python 3.9.0 documentation"](https://docs.python.org/3/howto/regex.html)

## Connect
If you enjoyed the blog post, feel free to connect with me on [Twitter](https://twitter.com/amitness) where I share new blog posts every week.