20 Small Steps to Become a Regex Master
Tags: regex, beginners, tutorial, firstyearincode
Published: 2019-11-09 12:19:06

> _"For the love of money is the root of all evil..."_
>
> -- [King James Bible, 1 Timothy 6:10](https://www.kingjamesbibleonline.org/1-Timothy-Chapter-6/#10)

Cover image by <a href="https://pixabay.com/users/msandersmusic-1972097/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=1181864">msandersmusic</a> from <a href="https://pixabay.com/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=1181864">Pixabay</a>

---

The above Bible verse is often taken out of context, with the first few words removed: "money is the root of all evil". The full quote, as seen above, clarifies that it's not money _itself_ which the author considers evil, but the _love_ of money. The surrounding verses emphasise that, so long as we are clothed and fed, what more is there to want? This small passage is a warning against greed, but it is also a lesson in intention, and the inherent moral neutrality of non-conscious things.

A common quip against using regular expressions ("RegEx" or just "regex") in programming is a quote attributed to Jamie Zawinsky:

> _"Some people, when confronted with a problem, think 'I know, I'll use regular expressions.' Now they have two problems."_ [[ source ]](https://blog.codinghorror.com/regular-expressions-now-you-have-two-problems/)

...but using regex isn't inherently a good idea or a bad idea. And it won't, by itself, add to your problems or solve any of them. It is a tool. And the way that you use it (properly or improperly) determines what kind of results you will see. If you try to use regex to build an HTML parser, [you're gonna have a bad time](https://www.youtube.com/watch?v=6Ls5j5iz2eA). But if you just want to extract some timestamps from some strings, you'll probably be okay.

In order to facilitate you having a _good_ time with regex, I've put together this tutorial, which gets you from zero to mastery in just twenty small steps. This guide mostly focuses on basic concepts of regular expressions, and only delves into more complex topics / language-specific extensions as necessary.

> _[**N. B.**](https://en.wikipedia.org/wiki/Nota_bene) Unless otherwise specified, all regular expressions below are Perl-Compatible Regular Expressions [(PCRE)](https://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions)._

---

<a name="toc"></a>
## Table of Contents

Step  1: [the purpose of regex](#step01)
Step  2: [square brackets `[]`](#step02)
Step  3: [escape sequences](#step03)
Step  4: [the "any" character `.`](#step04)
Step  5: [character ranges](#step05)
Step  6: [the "not" carat `^`](#step06)
Step  7: [character classes](#step07)
Step  8: [the asterisk `*` and the plus sign `+`](#step08)
Step  9: [the "optional" question mark `?`](#step09)
Step 10: [the "or" pipe `|`](#step10)
Step 11: [parentheses `()` for capturing groups](#step11)
Step 12: [define more specific matches first](#step12)
Step 13: [curly braces `{}` for defined repetition](#step13)
Step 14: [`\b`, the zero-width boundary character](#step14)
Step 15: [the "start of line" carat `^` and "end of line" dollar sign `$`](#step15)
Step 16: [non-capturing groups `(?:)`](#step16)
Step 17: [backreferences `\N` and named capturing groups](#step17)
Step 18: [lookaheads and lookbehinds](#step18)
Step 19: [conditionals](#step19)
Step 20: [recursion and further learning](#step20)

---

<a name="step01"></a>
## Step 1: the purpose of regex

Regular expressions are used to find patterns in text. That's it. The __pattern__ might be something as simple as the word "dog" in this sentence:

```python
The quick brown fox jumps over the lazy dog.
```

That regular expression looks like

```python
dog
```

...easy enough, yeah?

The pattern could also be _any word which contains an 'o'_. That regular expression might look like

```python
\w*o\w*
```

([You can try that regex out here.](https://regex101.com/r/Mf9L3O/2))

You can see that as the requirements for a "match" get more complex, the regular expression gets more complex as well. There is extra notation to specify groups of characters and matching repeated patterns, which I'll explain below.

But once we find a pattern in some text, what do we do with it? Well, modern regex _engines_ allow you to extract those substrings from the contained text, or remove them, or replace them with other text. Regular expressions are used for text parsing and manipulation.

We might extract things that look like IP addresses, then try to ping them; or we might extract names and email addresses and file them in a database. Or we might use regex to find sensitive information (like Social Security numbers or phone numbers) in emails, and alert the user that they may be putting themselves at risk. Regex really is a versatile tool that is easy to learn, but difficult to master:

> _"Just as there is a difference between playing a musical piece well and making music, there is a difference between knowing about regular expressions and really understanding them."_
>
> -- [Jeffrey E. F. Friedl](https://www.goodreads.com/work/quotes/570486-mastering-regular-expressions), Mastering Regular Expressions

[[ back to Table of Contents ]](#toc)

<a name="step02"></a>
## Step 2: square brackets `[]`

The easiest regular expressions to understand are those that simply look for a character-to-character match between the regex pattern and the target string, like:

```python
pattern: cat
string:  The cat was cut when it ran under the car.
matches:     ^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/4))

But we can also specify _alternative_ matches using _square brackets_:

```python
pattern: ca[rt]
string:  The cat was cut when it ran under the car.
matches:     ^^^                               ^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/5))

Open-and-close square brackets tell the regex engine to match _any one of the characters_ specified, but _only one_. The above regex won't -- for example -- do what you might expect with the following setup:

```python
pattern: ca[rt]
string:  The cat was cut when it ran under the cart.
matches:     ^^^                               ^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/6))

When you use square brackets, you're telling the regex engine to match on _exactly one_ of the characters contained within the brackets. If the engine finds a `c` character, then an `a` character, but the next character isn't `r` or `t`, it's not a match. If it finds `ca` and then _either_ `r` or `t`, it stops. It won't continue and try to match _more_ characters, because the square brackets indicate that _only one of the contained characters_ should be searched for. When it finds the `ca`, then the `r` in `cart`, it stops, because it's found a match on the sequence `car`.

### Pop Quiz:

Can you write a regular expression to match all ten `had`s and `Had`s in this passage?

```python
pattern: 
string:  Jim, where Bill had had "had", had had "had had". "Had had" had been correct.
matches:                 ^^^ ^^^  ^^^   ^^^ ^^^  ^^^ ^^^    ^^^ ^^^  ^^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/7))

What about all of the animal names in the following sentence?

```python
pattern:
string:  A bat, a cat, and a rat walked into a bar...
matches:   ^^^    ^^^        ^^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/8))

...or just the words `bar` and `bat`?

```python
pattern:
string:  A bat, a cat, and a rat walked into a bar...
matches:   ^^^                                 ^^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/9))

You're already writing more complex regular expressions and we're only at Step 2! Let's keep going!

[[ back to Table of Contents ]](#toc)

<a name="step03"></a>
## Step 3: escape sequences

In the previous Step, we learned about square brackets `[]` and how they help us to provide _alternative matches_ for the regex engine to find. But what if we want to match a literal open-and-close square bracket pair `[]`?

```python
You can't match [] using regex! You will regret this!
```

When we wanted a character-to-character match previously (like with the word `cat`), we would just type those characters exactly:

```python
pattern: []
string:  You can't match [] using regex! You will regret this!
matches: 
```

([See this in action here.](https://regex101.com/r/Mf9L3O/10))

This doesn't seem to work, though. This is because the square bracket characters `[` and `]` are _special characters_ that are usually used to denote _something other than_ a simple character-to-character match. As we saw in [Step 2](#step02), they're used to provide alternative matches so the regex engine can match _any one of the characters_ contained within them. If you don't put any characters in between them, this can cause an error.

To match these special characters, we must _escape_ them by preceding them with a backslash character `\`. The backslash character is _another_ special character that tells the regex engine to treat the next character _literally_, and not as a special character. By preceding both the `[` and the `]` characters with a `\` character, the regex engine will match each of them literally:

```python
pattern: \[\]
string:  You can't match [] using regex! You will regret this!
matches:                 ^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/11))

If we want to match a literal `\`, we can _escape_ it by preceding it with a second `\`:

```python
pattern: \\
string:  C:\Users\Tanja\Pictures\Dogs
matches:   ^     ^     ^        ^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/12))

_Only special characters_ should be preceded by `\` to force a literal match. All other characters are interpreted literally by default. For instance, the regular expression `t` matches only literal lowercase letter `t` characters:

```python
pattern: t
string:  t  t   t   t
matches: ^  ^   ^   ^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/14))

But the escape sequence `\t` is totally different. It matches _tab_ characters:

```python
pattern: \t
string:  t  t   t   t
matches:  ^  ^   ^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/13))

Other common escape sequences include `\n` (UNIX-style line breaks) and `\r` (used in Windows-style line breaks, `\r\n`). `\r` is the "carriage return" character and `\n` is the "line feed" character, [both of which were defined along with the ASCII standard when teletypes were still in common usage](https://stackoverflow.com/q/3091524/2925434).

Other common escape sequences will be covered later in this tutorial.

### Pop Quiz:

Can you match this regex `\[\]` with a regex? Your goal should be something like:

```python
pattern:  
string:  ...match this regex `\[\]` with a regex?
matches:                      ^^^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/15))

Can you match all the escape sequences in this example?

```python
pattern: 
string:  `\r`, `\t`, and `\n` are all regex escape sequences.
matches:  ^^    ^^        ^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/16))

[[ back to Table of Contents ]](#toc)

<a name="step04"></a>
## Step 4: the "any" character `.`

In writing your solutions to match the escape sequences we've seen so far, you may have been wondering... "can't I just match a backslash character and then _any_ other character following it?" Well, you can.

There's another special character which is used to match (nearly) _any_ character, and that's the period / full stop character `.`.

```python
pattern: .
string:  I'm sorry, Dave. I'm afraid I can't do that.
matches: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/17))

If you want to match only patterns that look like escape sequences, you could do something like:

```python
pattern: \\.
string:  Hi Walmart is my grandson there his name is "\n \r \t".
matches:                                              ^^ ^^ ^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/18))

And as with all special characters, if you want to match a literal `.`, you need to precede it with a `\` character:

```python
pattern: \.
string:  War is Peace. Freedom is Slavery. Ignorance is Strength.
matches:             ^                   ^                      ^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/19))

[[ back to Table of Contents ]](#toc)

<a name="step05"></a>
## Step 5: character ranges

What if you don't want to match _any_ character, though, but just letters? Or digits? Or vowels? Character classes and ranges allow us to achieve this.

```python
`\n`, `\r`, and `\t` are whitespace characters, `\.`, `\\` and `\[` are not.
```

Characters are "whitespace" [if they don't create any visible mark within text](https://en.wikipedia.org/wiki/Whitespace_character). A space character `' '` is whitespace, as is a line break, or a tab. Suppose we want to match the escape sequences representing the whitespace characters `\n`, `\r`, and `\t` in the above passage, but not the other escape sequences. How could we do that?

```python
pattern: \\[nrt]
string:  `\n`, `\r`, and `\t` are whitespace characters, `\.`, `\\` and `\[` are not.
matches:  ^^    ^^        ^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/20))

This works, but it's not very elegant. What if we later need to match the escape sequence for the "form feed" character, `\f`? (This character is used to indicate _page breaks_ in text.)

```python
pattern: \\[nrt]
string:  `\n`, `\r`, `\t`, and `\f` are whitespace characters, `\.`, `\\` and `\[` are not.
matches:  ^^    ^^    ^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/38))

With this approach we need to list, individually, every lowercase letter we want to match within the square brackets. An easier way of accomplishing this is to use _character ranges_ to match any lowercase letter:

```python
pattern: \\[a-z]
string:  `\n`, `\r`, `\t`, and `\f` are whitespace characters, `\.`, `\\` and `\[` are not.
matches:  ^^    ^^    ^^        ^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/39))

Character ranges work the way you might expect, given the example above. Put the first and last letters you want to match in the square brackets, with a hyphen in between them. If you only want to match the letters `a` through `m`, for instance, you could do:

```python
pattern: \\[a-m]
string:  `\n`, `\r`, `\t`, and `\f` are whitespace characters, `\.`, `\\` and `\[` are not.
matches:                        ^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/40))

If you want to match multiple ranges, just put them back-to-back in the square brackets:

```python
pattern: \\[a-gq-z]
string:  `\n`, `\r`, `\t`, and `\f` are whitespace characters, `\.`, `\\` and `\[` are not.
matches:        ^^    ^^        ^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/41))

Other common character ranges include: `A-Z` and `0-9`.

### Pop Quiz:

Hexadecimal numbers can contain digits `0-9` as well as letters `A-F`. When used to specify colours, "hex" codes can be as short as three characters. Create a regex to find valid hex codes in the list below:

```python
pattern:  
string:  1H8 4E2 8FF 0P1 T8B 776 42B G12
matches:     ^^^ ^^^         ^^^ ^^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/25))

Using character ranges, create a regex that will select only the lowercase consonants (non-vowel characters, including `y`) in the sentence below:

```
pattern:  
string:  The walls in the mall are totally, totally tall.
matches:  ^  ^ ^^^  ^ ^^  ^ ^^  ^  ^ ^ ^^^  ^ ^ ^^^ ^ ^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/26))

[[ back to Table of Contents ]](#toc)

<a name="step06"></a>
## Step 6: the "not" carat `^`

My solution for that last problem is a bit long. It took 17 characters to say "get the whole alphabet except the vowels". Surely there's an easier way to do this. As it turns out, there is.

The "not" carat `^` allows us to specify characters and character ranges which the regex engine should _not_ match on. An easier solution to the last pop quiz question above would be to match every character that's _not_ a vowel:

```
pattern: [^aeiou]
string:  The walls in the mall are totally, totally tall.
matches: ^^ ^^ ^^^^ ^^^^ ^^ ^^^ ^ ^^ ^ ^^^^^^ ^ ^^^^^ ^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/27))

The carat `^` as the leftmost character inside the square brackets `[]` tells the regex engine to match one single character which is _not_ within the square brackets. This means that the above regex also matches all spaces, the period `.`, the comma `,`, and the capital `T` at the beginning of the sentence. To exclude those, we can put them inside of the square brackets, as well:

```
pattern: [^aeiou .,T]
string:  The walls in the mall are totally, totally tall.
matches:  ^  ^ ^^^  ^ ^^  ^ ^^  ^  ^ ^ ^^^  ^ ^ ^^^ ^ ^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/28))

Note that we don't need to escape the `.` here. Many special characters within square brackets are treated literally, including the open `[` -- but not the close `]` bracket character (can you see why?). The backslash `\` character is also not treated literally. If you want to match on a literal backslash `\` using square brackets, you have to escape it by preceding it with a second backslash `\\`. This behaviour must be allowed in order for whitespace characters to be matchable within square brackets:

```python
pattern: [\t]
string:  t  t   t   t
matches:  ^  ^   ^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/29))

The carat can be used with ranges, as well. If I wanted to only capture the characters `a`, `b`, `c`, `x`, `y`, and `z`, I could do:

```python
pattern: [abcxyz]
string:  abcdefghijklmnopqrstuvwxyz
matches: ^^^                    ^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/31))


...or, I could specify that I want any character _not_ between `d` and `w`:

```python
pattern: [^d-w]
string:  abcdefghijklmnopqrstuvwxyz
matches: ^^^                    ^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/32))

Be careful with the "not" carat `^`. It's easy to think, "well, I said `[^b-f]`", so I should get a lowercase letter `a`, or something after `f`. That's not the case. That regex will match _any character_ not within that range, including digits, symbols, and whitespace.

```python
pattern: [^d-w]
string:  abcdefg h.i,j-klmnopqrstuvwxyz
matches: ^^^    ^ ^ ^ ^             ^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/33))

---

### Pop Quiz:

Use the "not" carat `^` within square brackets to match all of the words below that don't end with a `y`:

```python
pattern: 
string:  day dog hog hay bog bay ray rub
matches:     ^^^ ^^^     ^^^         ^^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/34))

Write a regex using a range and a "not" carat `^` to find all the years between 1977 and 1982 (inclusive) below:

```python
pattern: 
string:  1975 1976 1977 1978 1979 1980 1981 1982 1983 1984
matches:           ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/35))

Write a regex to match all characters below that aren't a literal carat `^` character:

```python
pattern: 
string:  abc1^23*()
matches: ^^^^ ^^^^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/36))

[[ back to Table of Contents ]](#toc)

<a name="step07"></a>
## Step 7: character classes

Even easier than character _ranges_ are character _classes_. Different regex engines have different available classes, so I'll only cover the basics here. (Check which version of regex you're using, because there may be more -- or different -- classes available than those shown here.)

Character classes work very similarly to _ranges_, but you can't specify the "start" and "end" values:

| class | characters |
| - | - |
| `\d` | "digits" `[0-9]` |
| `\w` | "word characters" `[A-Za-z0-9_]` |
| `\s` | "whitespace" `[ \t\r\n\f]` |


The `\w` word character class is particularly useful, as this set of characters is often required for _valid identifiers_ (variable and function names, etc.) in various programming languages.

We can use `\w` to simplify this regex that we saw previously:

```python
pattern: \\[a-z]
string:  `\n`, `\r`, `\t`, and `\f` are whitespace characters, `\.`, `\\` and `\[` are not.
matches:  ^^    ^^    ^^        ^^
```

With `\w`, we can instead write:

```python
pattern: \\\w
string:  `\n`, `\r`, `\t`, and `\f` are whitespace characters, `\.`, `\\` and `\[` are not.
matches:  ^^    ^^    ^^        ^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/42))

### Pop Quiz:

In the Java programming language, an _identifier_ (the name of a variable, class, function, etc.) must start with a letter `a-zA-Z`, a dollar sign `$`, or an underscore `_`. The remainder of the characters must be _word characters_ `\w`. Using one or more character classes, create a regex to find valid Java identifiers among the following 3-character sequences:

```python
pattern: 
string:  __e $12 .x2 foo Bar 3mm
matches: ^^^ ^^^     ^^^ ^^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/43))

United States Social Security Numbers (SSNs) are 9-digit numbers in the format `XXX-XX-XXXX`, where each `X` can be any digit `[0-9]`. Using one or more character classes, write a regex to find the properly-formatted SSNs in the list below:

```python
pattern: 
string:  113-25=1902 182-82-0192 H23-_3-9982 1I1-O0-E38B
matches:             ^^^^^^^^^^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/44))

[[ back to Table of Contents ]](#toc)

<a name="step08"></a>
## Step 8: the asterisk `*` and the plus sign `+`

So far, we've more or less gotten away with only matching on strings of a set length. But in the last pop quiz, we were getting close to the limit of what we can do with the notation we've seen so far.

Suppose, for instance, that we weren't limited to 3-character Java identifiers, but we could have identifiers of _any_ length. A solution which may have worked on the previous example _will not work_ for the following example:

```python
pattern: [a-zA-Z_$]\w\w
string:  __e $123 3.2 fo Barr a23mm ab x
matches: ^^^ ^^^         ^^^  ^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/48))

Notice how, when the identifier is valid, but longer than 3 characters, only the first three characters are matched. And when the identifier is valid, but fewer than 3 characters, it isn't matched at all!

The problem is that bracketed expressions `[]` match _exactly one_ character, as do the character classes like `\w`. This means any matches on the above regex _must be exactly three characters_ long. So it doesn't work as we might have hoped.

The special characters `*` and `+` can help here. These are modifiers which can be added to the right of any expression to match that expression more than once.

The Kleene star (or "asterisk"), `*`, will match the preceding token _any number of times_, including zero. The "plus sign" `+`, will match _one or more times_. So an expression which precedes a `+` is _mandatory_ (at least once), while an expression which precedes a `*` is optional, but when it does appear, it can appear any number of times.

With this knowledge, we can fix the above regex:

```python
pattern: [a-zA-Z_$]\w*
string:  __e $123 3.2 fo Barr a23mm ab x
matches: ^^^ ^^^^     ^^ ^^^^ ^^^^^ ^^ ^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/49))

We're now matching on valid identifiers of any length! Success!

What would have happened if we'd used `+` above instead of `*`?

```python
pattern: [a-zA-Z_$]\w+
string:  __e $123 3.2 fo Barr a23mm ab x
matches: ^^^ ^^^^     ^^ ^^^^ ^^^^^ ^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/50))

We dropped the last match, `x`. This is because `+` requires _at least one character to match_, but since the bracketed `[]` expression preceding `\w+` already "ate" the `x` character, there are no characters remaining, so the match fails.

When would we use `+`? When we want at least one match, but don't care how many times we match a given expression. For instance, maybe we want to match any numbers containing a decimal point:

```python
pattern: \d*\.\d+
string:  0.011 .2 42 2.0 3.33 4.000 5 6 7.89012
matches: ^^^^^ ^^    ^^^ ^^^^ ^^^^^     ^^^^^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/51))

Notice how -- by making numbers _to the left of_ the decimal point optional -- we were able to match both `0.011` and `.2`. But we required exactly one decimal point with `\.` and at least one digit to the right of the decimal point with `\d+`. The above regex _wouldn't_ match a number like `3.`, though, because we _require_ at least one digit to the right of the decimal point.

### Pop Quiz:

Match all of the English words in the passage below.

```python
pattern: 
string:  3 plus 3 is six but 4 plus three is 7
matches:   ^^^^   ^^ ^^^ ^^^   ^^^^ ^^^^^ ^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/52))

Match all of the file sizes in the list below. File sizes will be composed of a number (with or without a decimal point), followed by `KB`, `MB`, `GB`, or `TB`:

```python
pattern: 
string:  11TB 13 14.4MB 22HB 9.9GB TB 0KB
matches: ^^^^    ^^^^^^      ^^^^^    ^^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/53))

[[ back to Table of Contents ]](#toc)

<a name="step09"></a>
## Step 9: the "optional" question mark `?`

If you haven't yet, try to write a regex to solve that last pop quiz question. Did it work? Now try it here:

```python
pattern: 
string:  1..3KB 5...GB ..6TB
matches: 
```

Obviously, none of these are valid file sizes, so a good regex shouldn't match any of them. The solution I wrote for the last pop quiz question matches all of them, at least in part:

```python
pattern: \d+\.*\d*[KMGT]B
string:  1..3KB 5...GB ..6TB
matches: ^^^^^^ ^^^^^^   ^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/55))

What's the problem? We only really want _one_ decimal point, if any. But `*` allows any number of matches, including zero. Is there any way to only match zero times or once? But no more than once? There is.

The "optional" question mark `?` is a modifier that matches zero or one of the preceding characters, but no more:

```python
pattern: \d+\.?\d*[KMGT]B
string:  1..3KB 5...GB ..6TB
matches:    ^^^          ^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/56))

We're getting closer to a match here, but we're still not quite there. We'll see how to fix this in a few steps.

### Pop Quiz:

In some programming languages (like Java), "long integers" and floating-point numbers can be followed by `l`/`L` and `f`/`F` to indicate that they should be treated as `long`s / `float`s (respectively) rather than the usual `int`s / `doubles`. Find all of the valid `long`s in the line below:

```python
pattern: 
string:  13L long 2l 19 L lL 0
matches: ^^^      ^^ ^^      ^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/54))

[[ back to Table of Contents ]](#toc)

<a name="step10"></a>
## Step 10: the "or" pipe `|`

We had some difficulties earlier with matching various kinds of floating point numbers:

```python
pattern: \d*\.\d+
string:  0.011 .2 42 2.0 3.33 4.000 5 6 7.89012
matches: ^^^^^ ^^    ^^^ ^^^^ ^^^^^     ^^^^^^^
```

The above pattern matches numbers with a decimal point, and at least one digit to the right of the decimal point. But what if we also want to match strings like `0.`? (With no numbers to the right of the decimal point.)

We could write a regex like:

```python
pattern: \d*\.\d*
string:  0.011 .2 42 2.0 3.33 4.000 5 6 7.89012 0. .
matches: ^^^^^ ^^    ^^^ ^^^^ ^^^^^     ^^^^^^^ ^^ ^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/56))

That matches `0.`, but it also matches just a single `.`, as you can see above. Really, what we're trying to match on above are two different classes of strings:

1. numbers with at least one digit to the right of a decimal point, and
2. numbers with at least one digit to the left of a decimal point

These two regexes could be written independently as, respectively:

```python
pattern: \d*\.\d+
string:  0.011 .2 42 2.0 3.33 4.000 5 6 7.89012 0. .
matches: ^^^^^ ^^    ^^^ ^^^^ ^^^^^     ^^^^^^^
```

```python
pattern: \d+\.\d*
string:  0.011 .2 42 2.0 3.33 4.000 5 6 7.89012 0. .
matches: ^^^^^       ^^^ ^^^^ ^^^^^     ^^^^^^^ ^^
```

We can see that in neither case are the strings `42`, `5`, `6`, or `.` matched. What we want is the _union_ of these two regexes. How can we achieve that?

The "or" pipe `|` allows us to specify multiple possible match sequences in a regular expression. Similar to how `[]` let us specify alternate _single characters_, with the "or" pipe `|`, we can specify alternate multi-character expressions.

For instance, if we wanted to match either "dog" or "cat", we could write:

```python
pattern: \w\w\w
string:  Obviously, a dog is a better pet than a cat.
matches: ^^^^^^^^^    ^^^      ^^^^^^ ^^^ ^^^    ^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/57))

...but this matches _all_ three character sequences. "dog" and "cat" don't even have any letters in common, so we can't use square brackets to help us here. The easiest regex we could write which matches both of -- and only -- these two words, is:

```python
pattern: dog|cat
string:  Obviously, a dog is a better pet than a cat.
matches:              ^^^                        ^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/58))

The regex engine first attempts to match the entire sequence to the left of the pipe `|`, but if that fails, it then attempts to match the sequence to the right of the pipe. Multiple pipes can be chained together to match on more than two alternative sequences:

```python
pattern: dog|cat|pet
string:  Obviously, a dog is a better pet than a cat.
matches:              ^^^             ^^^        ^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/59))

### Pop Quiz:

Use the "or" pipe `|`, to fix the decimal regex given above:

```python
pattern: 
string:  0.011 .2 42 2.0 3.33 4.000 5 6 7.89012 0. .
matches: ^^^^^ ^^    ^^^ ^^^^ ^^^^^     ^^^^^^^ ^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/60))

Use the "or" pipe `|`, character classes, the "optional" question mark `?`, and more to create a single regex that matches _both_ long integers and floats, as discussed in the pop quiz at the end of the previous step (this is a really tough one):

```python
pattern: 
string:  42L 12 x 3.4f 6l 3.3 0F L F .2F 0.
matches: ^^^ ^^   ^^^^ ^^ ^^^ ^^     ^^^ ^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/61))

[[ back to Table of Contents ]](#toc)

<a name="step11"></a>
## Step 11: parentheses `()` for capturing groups

In that last pop quiz question, we were able to capture different kinds of integral and floating-point numerical values. But the regex engine made no distinction between those two kinds of values, since everything was captured in a single, big regular expression.

We can tell the regex engine to make a distinction between different _kinds_ of matches by surrounding them with parentheses:

```python
pattern: ([A-Z])|([a-z])
string:  The current President of Bolivia is Evo Morales.
matches: ^^^ ^^^^^^^ ^^^^^^^^^ ^^ ^^^^^^^ ^^ ^^^ ^^^^^^^
group:   122 2222222 122222222 22 1222222 22 122 1222222
```

([See this in action here.](https://regex101.com/r/Mf9L3O/62))

The above regex defines two _capturing groups_, which are indexed starting from 1. The first capturing group matches any single uppercase letter, and the second capturing group matches any single lowercase letter. Using the "or" pipe `|` and the "capturing group" parentheses `()` we can define a single regular expression which matches multiple kinds of strings.

If we apply this to our long/float regex above, the regex engine will capture the appropriate matches within the appropriate groups. By checking which group a string was matched into, we can tell immediately whether it's a float value or a long value:

```python
pattern: (\d*\.\d+[fF]|\d+\.\d*[fF]|\d+[fF])|(\d+[lL])
string:  42L 12 x 3.4f 6l 3.3 0F L F .2F 0.
matches: ^^^      ^^^^ ^^     ^^     ^^^
group:   222      1111 22     11     111
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/63))

This regular expression is pretty complex, but you should now be able to understand every part of it. Let's break it apart so we can review each of these symbols:

```
(                // match any "float" string
  \d*\.\d+[fF]
  |
  \d+\.\d*[fF]
  |
  \d+[fF]
)
|               // OR
(               // match any "long" string
  \d+[lL]
)
```

The "or" pipe `|` and parenthetical capturing groups `()` allow us to match on different kind of strings. In this case, we're matching _either_ "float" floating-point numbers or "long" long integer numbers.

```
(
  \d*\.\d+[fF]  // 1+ digits to the right of the decimal point
  |
  \d+\.\d*[fF]  // 1+ digits to the left of the decimal point
  |
  \d+[fF]       // no decimal point, only 1+ digits
)
|
(
  \d+[lL]       // no decimal point, only 1+ digits
)
```

Within the "float" capturing group, we have three options -- numbers with at least 1 digit to the right of the decimal point, numbers with at least one digit to the left of the decimal point, and numbers with no decimal point. Any of these are "floats", provided they have an `f` or an `F` appended to the end.

Within the "long" capturing group, we only have a single option -- we must have 1 or more digits, followed by an `l` or an `L` character.

The regex engine will look for these substrings within the given string, and index them within the appropriate capturing group.

Note that we don't match on any numbers which _don't_ have one of `l`, `L`, `f`, or `F` appended. What should these numbers be categorised as? Well, if they have a decimal point, the default is `double` in the Java language. Otherwise, they should be `int`s.

### Pop Quiz:

Add two more capturing groups to the above regex so that it also categorises numbers as `double` or `int`. (This is another tough one, don't be discouraged if it takes a while or you need to peek at my solution.)

```python
pattern: 
string:  42L 12 x 3.4f 6l 3.3 0F L F .2F 0.
matches: ^^^ ^^   ^^^^ ^^ ^^^ ^^     ^^^ ^^
group:   333 44   1111 33 222 11     111 22
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/64))

Here's a slightly easier one. Use parenthetical capturing groups `()`, the "or" pipe `|`, and character ranges to sort the following ages into "legal to drink in the U.S." (>= 21) and "illegal to drink in the U.S." (< 21) groups:

```python
pattern: 
string:  7 10 17 18 19 20 21 22 23 24 30 40 100 120
matches: ^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^^
group:   2 22 22 22 22 22 11 11 11 11 11 11 111 111
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/65))

[[ back to Table of Contents ]](#toc)

<a name="step12"></a>
## Step 12: define more specific matches first

You may have had some trouble with that last pop quiz question if you tried to define "legal drinkers" as the _first_ capturing group rather than the second. To see why, let's look at a different example. Suppose we want to capture -- separately -- surnames with fewer than 4 characters and surnames with 4 or more characters. If we make the shorter names the _first_ capturing group, watch what happens:

```python
pattern: ([A-Z][a-z]?[a-z]?)|([A-Z][a-z][a-z][a-z]+)
string:  Kim Jobs Xu Cloyd Mohr Ngo Rock
matches: ^^^ ^^^  ^^ ^^^   ^^^  ^^^ ^^^
group:   111 111  11 111   111  111 111
```

([See this in action here.](https://regex101.com/r/Mf9L3O/66))

By default, most regex engines use [_greedy matching_](https://stackoverflow.com/a/2301298/2925434) on the basic characters we've seen so far. What that means is that a regex engine will capture the longest possible group, defined as early as possible within the provided regular expression. So even though the second group above _could have captured more characters_ in names like "Jobs" and "Cloyd", for instance, since the first three characters of these names were already captured by the first capturing group, they can't be captured again by the second one.

This is a simple fix, though -- just switch the order of the capturing groups, putting the more specific (longer) one first:

```python
pattern: ([A-Z][a-z][a-z][a-z]+)|([A-Z][a-z]?[a-z]?)
string:  Kim Jobs Xu Cloyd Mohr Ngo Rock
matches: ^^^ ^^^^ ^^ ^^^^^ ^^^^ ^^^ ^^^
group:   222 1111 22 11111 1111 222 111
```

([See this in action here.](https://regex101.com/r/Mf9L3O/67))

### Pop Quiz:

"More specific" almost always means "longer". Suppose we want to capture two kinds of "words": those that begin with vowels (more specific) vs. those that don't (any other word). How could we write a regex to capture and identify strings that match those two groups? (Groups below are _lettered_ rather than _numbered_. You must determine which group should be matched first.)

```python
pattern: 
string:  pds6f uub 24r2gp ewqrty l ui_op
matches: ^^^^^ ^^^ ^^^^^^ ^^^^^^ ^ ^^^^^
group:   NNNNN VVV NNNNNN VVVVVV N VVVVV
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/68))

In general, the more precise your regular expression is, the longer it will be. And the more precise it is, the less likely it is that you'll capture something you don't want. So even though they can look intimidating, longer regexes ~= better regexes. [Unfortunately.](https://emailregex.com/)

[[ back to Table of Contents ]](#toc)

<a name="step13"></a>
## Step 13: curly braces `{}` for defined repetition

In the surnames example from the previous step, we had a pretty repetitive-looking regular expression:

```python
pattern: ([A-Z][a-z][a-z][a-z]+)|([A-Z][a-z]?[a-z]?)
string:  Kim Jobs Xu Cloyd Mohr Ngo Rock
matches: ^^^ ^^^^ ^^ ^^^^^ ^^^^ ^^^ ^^^
group:   222 1111 22 11111 1111 222 111
```

For the first group, we wanted surnames with four or more letters. The second group was meant to capture surnames with three or fewer letters. Is there not a simpler way to write this, rather than repeating those `[a-z]` groups over and over again? There is, using curly braces `{}`.

Curly braces `{}` allow us to specify a minimum and (optionally) maximum number of times the preceding character or capturing group should be matched. There are three possibilities with `{}`:

```
{X}   // match exactly X times
{X,}  // match >= X times
{X,Y} // match >= X and <= Y times
```

Here are examples of those three different syntaxes:

```python
pattern: [a-z]{11}
string:  humuhumunukunukuapua'a
matches: ^^^^^^^^^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/69))

```python
pattern: [a-z]{18,}
string:  humuhumunukunukuapua'a
matches: ^^^^^^^^^^^^^^^^^^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/71))

```python
pattern: [a-z]{11,18}
string:  humuhumunukunukuapua'a
matches: ^^^^^^^^^^^^^^^^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/70))

There are a few things to notice in the above examples. First, using the `{X}` notation, the preceding character or group will be matched exactly that number (`X`) of times. If there are _more_ characters that _could have matched_ if `X` were greater (as shown in the first example), they will not be included in the match. If there are _fewer_ than `X` characters, the entire match fails (try changing `11` to `99` in the first example).

Second, both the `{X,}` and the `{X,Y}` notations are _greedy_. They will match _as many characters as they can_ while still satisfying the defined regular expression. If you say `{3,7}` -- between 3 and 7 characters can be matched -- and the next 7 characters are valid, then all 7 characters will be matched. If you say `{1,}`, but the next 14,000 characters all match, then all 14,000 of those characters will be included in the matched string.

So how can we use this to rewrite our expression above? A really simple improvement could be to replace the adjacent `[a-z]` groups with `[a-z]{N}`, where `N` is appropriately chosen:

```python
pattern: ([A-Z][a-z]{2}[a-z]+)|([A-Z][a-z]?[a-z]?)
```

...but that doesn't make it all that much nicer. Look at the first capturing group: we have `[a-z]{2}` (matches exactly 2 lowercase letters) followed by `[a-z]+` (matches 1 or more lowercase letters). We can simplify this by asking for 3 or more lowercase letters, using curly braces:

```python
pattern: ([A-Z][a-z]{3,})|([A-Z][a-z]?[a-z]?)
```

The second capturing group is different. We want _at most_ three characters in these surnames, which means we have an upper limit, but our lower limit is zero:

```python
pattern: ([A-Z][a-z]{3,})|([A-Z][a-z]{0,2})
```

Now, specificity is always better when using regexes, so we would be wise to stop here, but I can't help but notice that those two character ranges (`[A-Z]` and `[a-z`]) right next to each other almost look like the "word character" class, `\w` (`[A-Za-z0-9_]`). If we're sure that our data _only_ contains nicely-formatted surnames, we could simplify our regex and write just:

```python
pattern: (\w{4,})|(\w{1,3})
```

The first group captures any sequence of 4 or more word characters (`[A-Za-z0-9_]`) and the second group captures any sequence of 1 to 3 word characters (inclusive). Does it work?

```python
pattern: (\w{4,})|(\w{1,3})
string:  Kim Jobs Xu Cloyd Mohr Ngo Rock
matches: ^^^ ^^^^ ^^ ^^^^^ ^^^^ ^^^ ^^^
group:   222 1111 22 11111 1111 222 111
```

([See this in action here.](https://regex101.com/r/Mf9L3O/72))

It does! How about that. And so much cleaner than our original example. Since the first capturing group matches all surnames with four or more characters, we could even change the second capturing group to just `\w+`, since this would capture all remaining surnames (those with 1, 2, or 3 characters):

```python
pattern: (\w{4,})|(\w+)
string:  Kim Jobs Xu Cloyd Mohr Ngo Rock
matches: ^^^ ^^^^ ^^ ^^^^^ ^^^^ ^^^ ^^^
group:   222 1111 22 11111 1111 222 111
```

([See this in action here.](https://regex101.com/r/Mf9L3O/73))

Concise!

### Pop Quiz:

Use curly braces `{}` to rewrite the Social Security Number regex from [Step 7](#step07):

```python
pattern: 
string:  113-25=1902 182-82-0192 H23-_3-9982 1I1-O0-E38B
matches:             ^^^^^^^^^^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/74))

Suppose a password strength verification system on a website requires user passwords to be between 6 and 12 non-whitespace characters. Write a regex that flags bad passwords in the list below. Each password is contained within parentheses `()` for easy _regexing_, so make sure your regex starts and ends with literal `(` and `)` characters. (Hint: make sure you _disallow_ literal parentheses in the passwords with `[^()]` or similar, otherwise you might end up matching the entire line!)

```python
pattern: 
string:  (12345) (my password) (Xanadu.2112) (su_do) (OfSalesmen!)
matches: ^^^^^^^ ^^^^^^^^^^^^^               ^^^^^^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/75))

[[ back to Table of Contents ]](#toc)

<a name="step14"></a>
## Step 14: `\b`, the zero-width boundary character

The last pop quiz question was tough. But what if we made it _even more difficult_ by surrounding the passwords with quotes `""` instead of parentheses `()`? Could we write a similar solution by simply replacing all of the literal `(` and `)` characters with `"`?

```python
pattern: \"[^"]{0,5}\"|\"[^"]+\s[^"]*\"
string:  "12345" "my password" "Xanadu.2112" "su_do" "OfSalesmen!"
matches: ^^^^^^^ ^^^^^^^^^^^^^             ^^^     ^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/76))

This fails pretty spectacularly. Can you see why?

The problem is that we're looking for _bad passwords_ here. `"Xanadu.2112"` is a _good password_, so when the regex realizes that it doesn't contain any spaces or literal `"` characters, it gives up, _just before_ the `"` character which bounds the password on the right-hand side. (Because we specified that `"` characters can't be found within passwords, using `[^"]`.)

Once the regex engine is sure that those characters don't match the defined regex, it starts again, at exactly the place it left off -- which is the `"` which bounds `"Xanadu.2112"` on the right. From there, it sees a single space character, and another `"` -- a bad password! So it matches `" "`, and continues.

It would be really nice if we could specify that the _first character_ of a password must be non-whitespace. Is there a way to do that? (You should know by now that the answer to all of my rhetorical questions is "yes".) Yes! There is!

Many regex engines provide the "word boundary" escape sequence `\b`. `\b` is a _zero-width_ escape sequence which matches, funnily enough, the _boundary_ of a word. Remember that when we say "word", we mean any sequence of characters in the class `\w`, aka. `[a-zA-Z0-9_]`.

Matching on a word boundary means that the character immediately before or immediately after the `\b` sequence must be a non-word character. But we don't actually _include_ that character in our captured string. To see how this works, let's look at a small example:

```python
pattern: \b[^ ]+\b
string:  Ve still vant ze money, Lebowski.
matches: ^^ ^^^^^ ^^^^ ^^ ^^^^^  ^^^^^^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/78))

The sequence `[^ ]` should match _any character_ that's not the literal space character. So why doesn't it match the `,` after `money` or the `.` after `Lebowski`? It's because `,` and `.` are not word characters, so there are _boundaries_ created between word characters and non-word characters. These appear between the `y` of `money` and the `,` that follows it and between the `i` of `Lebowski` and the full stop / period which follows it. The regex matches on those word boundaries (but not the non-word characters which help to define them).

What would happen if we _didn't_ include the `\b` sequence?

```python
pattern: [^ ]+
string:  Ve still vant ze money, Lebowski.
matches: ^^ ^^^^^ ^^^^ ^^ ^^^^^^ ^^^^^^^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/79))

Aha, now we _do_ match those punctuation marks.

So now let's use word boundaries to help fix our quoted passwords regex:

```python
pattern: \"\b[^"]{0,5}\b\"|\"\b[^"]+\s[^"]*\b\"
string:  "12345" "my password" "Xanadu.2112" "su_do" "OfSalesmen!"
matches: ^^^^^^^ ^^^^^^^^^^^^^               ^^^^^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/80))

By placing word boundaries "inside" the quotes (`"\b...\b"`), we're effectively saying that the first and last characters of the matched passwords must be "word" characters. So this works fine here, but won't work as nicely if the first or last character of a user's password is not a word character:

```python
pattern: \"\b[^"]{0,5}\b\"|\"\b[^"]+\s[^"]*\b\"
string:  "thefollowingpasswordistooshort" "C++"
matches: 
```

([See this in action here.](https://regex101.com/r/Mf9L3O/81))

See how the second password is not flagged as "invalid", even though it's clearly too short? You need to be careful with `\b` sequences, since they only match boundaries between `\w` and non-`\w` characters. In the above example, because we allowed non-`\w` characters in passwords, the boundary between the `"` and the first/last character of the password _is not guaranteed_ to be a word boundary, `\b`.

### Pop Quiz:

Word boundaries are useful in _syntax highlighting_ engines, where we want to match on a particular sequence of characters, but we want to ensure that they only occur at the beginning or the end of a word (or by themselves entirely). Suppose we are writing a syntax highlighter and we want to highlight the word `var`, but only when it appears on its own (not touching any other word characters). Can you write a regex to do that?

```python
pattern: 
string:  var varx _var (var j) barvarcar *var var-> {var}
matches: ^^^            ^^^               ^^^ ^^^    ^^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/82))

[[ back to Table of Contents ]](#toc)

<a name="step15"></a>
## Step 15: the "start of line" carat `^` and "end of line" dollar sign `$`

The `\b` word boundary sequence from the last Step is not the only _zero-width_ special sequence available for use in regular expressions. Two of the more popular ones include the "start of line" carat `^` and the "end of line" dollar sign `$`. Including one of these in your regular expressions means that the given match _must appear_ at the beginning or end of a line within the string you're trying to match on:

```python
pattern: ^start|end$
string:  start end start end start end start end
matches: ^^^^^                               ^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/83))


If your string includes line breaks, then `^start` will match the sequence `start` at the beginning of _any_ line and `end$` will match the sequence `end` at the end of any line (though those are difficult to show here). These characters are particularly useful when working with _delimited_ data.

Let's revisit the "file size" problem from [Step 9](#step09), using the "start of line" carat. In this example, our file sizes are delimited (separated) by space characters '&nbsp;'. So we want every file size to begin with a digit that's preceded by a space character _or the beginning of a line_:

```python
pattern: (^| )(\d+|\d+\.\d+)[KMGT]B
string:  6.6KB 1..3KB 12KB 5G 3.3MB KB .6.2TB 9MB
matches: ^^^^^       ^^^^^   ^^^^^^          ^^^^
groups:  222         122     1222            12
```

([See this in action here.](https://regex101.com/r/Mf9L3O/84))

We're _so close_! You can see we still have one small problem, where we're matching on the space character before valid file sizes. Now, we could just ignore that capturing group (`1`), when our regex engine finds it, or we could use a _non-capturing group_, which we'll see in the next Step.

### Pop Quiz:

Continuing our syntax highlighting example from the last Step, some syntax highlighters will mark _trailing spaces_ -- that is, any whitespace which comes between a non-whitespace character and the end of the line. Can you write a regex highlighting rule for trailing spaces?

```python
pattern: 
string:  myvec <- c(1, 2, 3, 4, 5)       
matches:                          ^^^^^^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/85))

A simple Comma-Separated Values (CSV) parser will look for "tokens", separated by commas. Generally, whitespace is not significant unless it's inside of quotes `""`. Can you write a simple CSV-parsing regex which matches tokens between commas but ignores (doesn't capture) non-quoted whitespace?

```python
pattern: 
string:  a, "b", "c d",e,f,   "g h", dfgi,, k, "", l
matches: ^^ ^^^^ ^^^^^^^^^^   ^^^^^^ ^^^^^^ ^^ ^^^ ^
groups:  21 2221 2222212121   222221 222211 21 221 2
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/114))

[[ back to Table of Contents ]](#toc)

<a name="step16"></a>
## Step 16: non-capturing groups `(?:)`

In two examples in the previous Step, we captured text where we really didn't need to. In the "file sizes" challenge, we grabbed the space characters before the first digit of the file sizes, and in the "CSV" challenge, we captured the commas between each token. We don't need to capture these characters, but we _do_ need to use them to structure our regular expression. These are perfect use cases for the _non-capturing group_, `(?:)`.

A non-capturing group does exactly what it sounds like -- it allows you to group characters, and use them in your regular expressions, but it doesn't _capture them_ within a numbered group:

```python
pattern: (?:")([^"]+)(?:")
string:  I only want "the text inside these quotes".
matches:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
groups:               1111111111111111111111111111
```

([See this in action here.](https://regex101.com/r/Mf9L3O/87))

Now, the _regular expression_ matched the text in the quotes, as well as the quote characters themselves, but _the capturing group_ only captured the text within the quotes. Why would we ever want to do this?

Well, most regex engines allow you to recover the text from the capturing groups defined within your regexes. If we can trim off the extra characters that we don't need, by not including them in our capturing groups, it makes it easier to parse and manipulate the text later.

Here's another example, cleaning up the CSV parser from the previous Step:

```python
pattern: (?:^|,)\s*(?:\"([^",]*)\"|([^", ]*))
string:  a, "b", "c d",e,f,   "g h", dfgi,, k, "", l
matches: ^   ^    ^^^  ^ ^     ^^^   ^^^^   ^      ^
groups:  2   1    111  2 2     111   2222   2      2
```

([See this in action here.](https://regex101.com/r/Mf9L3O/115))

There are a few things to note here. First, we no longer _capture_ the comma delimiters, since we changed the `(^|,)` capturing group to a `(?:^|,)` non-capturing group. Second, we've _nested a capturing group inside of a non-capturing group_. This is useful when, for instance, you need a group of characters to appear in a particular order, but you only care about a subset of those characters.

In our case, we needed non-quote, non-comma characters `[^",]*` to appear _within_ quotes, but we don't actually care about the quote characters themselves, so there was no need to capture them.

Finally, note that there's also a _zero-length match_ in the above example, in between the `k` and `l` characters. The `""` there _is a matched substring_, but there are no characters between the quotation marks (which we don't capture), so the matched substring contains no characters (has length zero).

### Pop Quiz:

Using non-capturing groups (and capturing groups, and character classes, etc.), write a regex which captures only the correctly-formatted file sizes in the string below:

```python
pattern: 
string:  6.6KB 1..3KB 12KB 5G 3.3MB KB .6.2TB 9MB
matches: ^^^^^       ^^^^^   ^^^^^^          ^^^^
groups:  11111        1111    11111           111
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/89))

Opening HTML _tags_ begin with a `<` character and end with a `>` character. Closing HTML tags begin with a `</` character sequence and end with a `>` character. The _name_ of the tag is contained within those characters. Can you write a regex to capture _only the names_ in  the following tags? (You _may_ be able to get away with solving this without using non-capturing groups. Try solving it two ways! Once using capturing groups and once without.)

```python
pattern: 
string:  <p> </span> <div> </kbd> <link>
matches: ^^^ ^^^^^^  ^^^^^ ^^^^^ ^^^^^^
groups:   1    1111   111    111   1111
```

([Here's a solution _with_ non-capturing groups.](https://regex101.com/r/Mf9L3O/92))

([Here's a solution _without_ non-capturing groups.](https://regex101.com/r/Mf9L3O/91))

[[ back to Table of Contents ]](#toc)

<a name="step17"></a>
## Step 17: backreferences `\N` and named capturing groups

Even though I warned you in the introduction that trying to build an HTML parser with regex usually leads to heartache, the last example is a good segue into another (sometimes) useful feature of most regexes: backreferences.

Backreferences are similar to repeated groups, in that you can try to capture the same text twice. But they differ in one important aspect -- they will only capture _the exact same text_, character-for-character.

So while a repeated group would allow us to capture something like

```python
pattern: (he(?:[a-z])+)
string:  heyabcdefg hey heyo heyellow heyyyyyyyyy
matches: ^^^^^^^^^^ ^^^ ^^^^ ^^^^^^^^ ^^^^^^^^^^^
groups:  1111111111 111 1111 11111111 11111111111
```

([See this in action here.](https://regex101.com/r/Mf9L3O/93))

...a backreference would only match

```python
pattern: (he([a-z])(\2+))
string:  heyabcdefg hey heyo heyellow heyyyyyyyyy
matches:                              ^^^^^^^^^^^
groups:                               11233333333
```

([See this in action here.](https://regex101.com/r/Mf9L3O/116))

Repeated capturing groups are useful for when you want to match the same _pattern_ repeatedly, while backreferences are good for when you want to match the _exact same_ text. For instance, we could use a backreference to try to find matching open-and-close HTML tags:

```python
pattern: <(\w+)[^>]*>[^<]+<\/\1>
string:  <span style="color: red">hey</span>
matches: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
groups:   1111
```

([See this in action here.](https://regex101.com/r/Mf9L3O/95))

Please note that this is an _extremely over-simplified_ example and I strongly recommend you do not try to write a regex-based HTML parser. It's a very complex syntax and you'll likely have a bad time.

__Named capturing groups__ are very similar to backreferences, so I'll briefly cover them here as well. The only difference between a backreference and a named capturing group is that... a named capturing group is named:

```python
pattern: <(?<tag>\w+)[^>]*>[^<]+<\/(?P=tag)>
string:  <span style="color: red">hey</span>
matches: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
groups:   1111
```

([See this in action here.](https://regex101.com/r/Mf9L3O/103))

You can create a named capturing group with the `(?<name>...)` or `(?'name'...)` syntax (.NET-compatible regex) or with the `(?P<name>...)` or `(?P'name'...)` (Python-compatible regex) syntax. Since we're using PCRE (Perl-compatible regex), which supports both versions, we can use either here.

To repeat a named capturing group later in the regex, we use `\k<name>` or `\k'name'` (.NET) or `(?P=name)` (Python). Again, PCRE supports all of these different varieties. [You can read more about named capturing groups here,](https://www.regular-expressions.info/named.html) but that's most of what you really need to know about them.

### Pop Quiz:

Use backreferences to help me remember... uh... that person's name.

```python
pattern: 
string:  "Hi my name's Joe." [later] "What's that guy's name? Joe?"
matches:        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
groups:                111
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/96))

[[ back to Table of Contents ]](#toc)

<a name="step18"></a>
## Step 18: lookaheads and lookbehinds

We're getting into some more advanced features of regular expressions now. Everything up to and including [Step 16](#step16), I use on a fairly regular basis. But these last few Steps are only for people who use regex very seriously to match very complex expressions. In other words, regex masters.

Lookaheads and lookbehinds may seem quite complex the first time you see them, but they're actually not too difficult. They allow you to do something similar to what we were doing with non-capturing groups earlier -- checking if some text exists immediately before or immediately after the actual text that we want to match. For instance, suppose we only want to match the names of things that people love, but only if they're very enthusiastic about it (only if they end their sentence with an exclamation point). We could do something like:

```python
pattern: (\w+)(?=!)
string:  I like desk. I appreciate stapler. I love lamp!
matches:                                           ^^^^
groups:                                            1111
```

([See this in action here.](https://regex101.com/r/Mf9L3O/97))

You can see how the above capturing group `(\w+)`, which would normally match on any of the words in the passage, _only_ matches the word `lamp`. The _positive lookahead_ `(?=...)` means that we can _only_ match sequences which end in a `!` character, but we do not actually match that exclamation point character itself. This is an important distinction, because with non-capturing groups, we match the character, but don't capture it. With lookaheads and lookbehinds, we use the character to build our regex, but then we do not even match on the character. We're free to match on it later in our regular expression.

There are four kinds of lookaheads and lookbehinds, in total: the positive lookahead `(?=...)`, the negative lookahead `(?!...)`, the positive lookbehind `(?<=...)`, and the negative lookbehind `(?<!...)`. They do what they sound like -- positive lookahead and lookbehind will allow the regex engine to continue matching only when the text contained within the lookahead/lookbehind _does_ match. Negative lookahead and lookbehind do the opposite -- they only allow the regex to match when the text contained within the lookahead/lookbehind _does not match_.

For example, we might want to only match method names in a chained sequence of methods, and not the object on which they're operating. In this case, each method name should be preceded by a literal `.` character. A regex using a simple lookbehind could help here:

```python
pattern: (?<=\.)(\w+)
string:  myArray.flatMap.aggregate.summarise.print
matches:         ^^^^^^^ ^^^^^^^^^ ^^^^^^^^^ ^^^^^
groups:          1111111 111111111 111111111 11111
```

([See this in action here.](https://regex101.com/r/Mf9L3O/98))

In the above text, we match any sequence of word characters `\w+`, but only if they're preceded by a literal `.`. We could have achieved something similar using non-capturing groups, but it's a little messier:

```python
pattern: (?:\.)(\w+)
string:  myArray.flatMap.aggregate.summarise.print
matches:        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
groups:          1111111 111111111 111111111 11111
```

([See this in action here.](https://regex101.com/r/Mf9L3O/99))

Even though it's shorter, it's matching on characters that we don't want. Although this example may seem trivial, lookaheads and lookbehinds can really help us clean up our regular expressions.

### Pop Quiz:

The negative lookbehind `(?<!...)` will only allow the regex engine to continue attempting to find a match if the text contained within the negative lookbehind _does not appear_ prior to the remainder of the text to be matched. For example, we might want to use a regex to match _only_ the surnames of the women attending a conference. To do that, we would want to make sure that a person's surname is _not_ preceded by a `Mr.`. Can you write a regex to do that? (You can assume surnames are at least four characters long.)

```python
pattern: 
string:  Mr. Brown, Ms. Smith, Mrs. Jones, Miss Daisy, Mr. Green
matches:                ^^^^^       ^^^^^       ^^^^^
groups:                 11111       11111       11111
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/100))

Suppose we're cleaning a database and we have a column of information which is meant to be a percentage. Unfortunately, some people have written the numbers as decimal values on the range [0.0, 1.0], while others have written percentages on the range [0.0%, 100.0%], and still others have written the percentage values, but have forgotten the literal percent sign `%`. Using a negative lookahead `(?!...)`, can you flag only the values which are meant to be percentages, but which are missing their `%` signs? These should be values strictly greater than 1.00, but without a trailing `%`. (No number has more than two digits before or after the decimal point.)

> Note that this solution is _extremely difficult_. If you can solve this without peeking at my answer, you have formidable regex skills.

```python
pattern: 
string:  0.32 100.00 5.6 0.27 98% 12.2% 1.01 0.99% 0.99 13.13 1.10
matches:      ^^^^^^ ^^^                ^^^^            ^^^^^ ^^^^
groups:       111111 111                1111            11111 1111
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/101))

[[ back to Table of Contents ]](#toc)

<a name="step19"></a>
## Step 19: conditionals

We're now at the point at which most people would stop using regular expressions. We've covered probably 95% of the use cases for simple regexes, and anything done in Steps 19 and 20 are usually performed by a more full-featured text manipulation language like `awk` or `sed` (or a general-purpose programming language). Still, let's carry on, just so you know what regex is really capable of.

Although [regular expressions are not Turing-complete](https://stackoverflow.com/questions/7983115/are-perl-regexes-turing-complete), some regex engines offer features which get awfully close to looking like a full programming language. One such feature is the _conditional_. Regex conditionals allow _if-then-else_ statements, where the branch that's taken is determined by either a _lookahead_ or a _lookbehind_, which we learned about in the last Step.

For instance, you might want to match only valid entries in a list of dates:

```python
pattern: (?<=Feb )([1-2][0-9])|(?<=Mar )([1-2][0-9]|3[0-1])
string:  Dates worked: Feb 28, Feb 29, Feb 30, Mar 30, Mar 31
matches:                   ^^      ^^              ^^      ^^
groups:                    11      11              22      22
```

([See this in action here.](https://regex101.com/r/Mf9L3O/104))

> Notice how the groups above are also indexed according to month. We could write a regex for all 12 months and capture only valid dates, which would then be captured into groups indexed by the month of the year.

The above uses a sort of _if_-like structure in that it will only match in the first group if `"Feb "` precedes the number (and similarly for the second). But what if we _only_ wanted special treatment for February? Something like "if the number is preceded by `"Feb "`, do this, _else_ do this other thing". That's what conditionals do:

```python
pattern: (?(?<=Feb )([1-2][0-9])|([1-2][0-9]|3[0-1]))
string:  Dates worked: Feb 28, Feb 29, Feb 30, Mar 30, Mar 31
matches:                   ^^      ^^              ^^      ^^
groups:                    11      11              22      22
```

([See this in action here.](https://regex101.com/r/Mf9L3O/105))

The _if-then-else_ structure looks like `(?(if)then|else)`, where `(if)` is replaced by a lookahead or lookbehind. In the example above, `(if)` is `(?<=Feb )`. You can see that we matched on dates greater than 29, but only if they didn't follow `"Feb "`. Using lookbehinds in conditionals is useful when you want to ensure a match is preceded by some text.

Positive lookahead conditionals can be confusing, because the conditional itself doesn't match any text. So if you want the _if_ clause to ever evaluate, it needs to be matchable from the lookahead, as shown below:

```python
pattern: (?(?=exact)exact|else)wo
string:  exact else exactwo elsewo
matches:            ^^^^^^^ ^^^^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/106))

This means that positive lookahead conditionals are kind of useless. You're checking to see if that text is ahead, and then providing a match template to follow when it is. The conditional isn't helping us at all. You might as well just replace the above with a simpler regex:

```python
pattern: (?:exact|else)wo
string:  exact else exactwo elsewo
matches:            ^^^^^^^ ^^^^^^
```

([See this in action here.](https://regex101.com/r/Mf9L3O/107))

So, rule of thumb with conditionals: test, test, test. Things that you may think are obvious will fail in exciting and unexpected ways.

### Pop Quiz:

Write a regular expression which uses a negative lookahead conditional to check if the next word starts with a capital letter. If it does, only capture a single capital letter, followed by lowercase letters. If it doesn't, capture any word characters.

```python
pattern: 
string:  Jones Smith 9sfjn Hobbes 23r4tgr9h CSV Csv vVv
matches: ^^^^^ ^^^^^ ^^^^^ ^^^^^^ ^^^^^^^^^     ^^^ ^^^
groups:  22222 22222 11111 222222 111111111     222 111
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/108))

Write a negative lookbehind conditional which only captures the text `owns` when it is _not_ preceded by the text `cl`, and which only captures the text `ouds` when it _is_ preceded by the text `cl`. (A bit of a contrived example, but what can you do.)

```python
pattern: 
string:  Those clowns owns some clouds. ouds.
matches:              ^^^^        ^^^^
```

([See one possible solution here.](https://regex101.com/r/Mf9L3O/109))

[[ back to Table of Contents ]](#toc)

<a name="step20"></a>
## Step 20: recursion and further learning

There is only so much that can be squeezed into a 20-step introduction to any topic, really, and regular expressions are no exception. There are many different [implementations of and standards for regular expressions](https://en.wikipedia.org/wiki/Regular_expression#Standards), which you can find peppered around the Internet. If you're interested in learning more, I suggest you check out the wonderful site [regularexpressions.info](https://www.regular-expressions.info/), it's a fantastic reference and I've certainly learned a lot about regex from it. I strongly recommend it, as well as [regex101.com](https://regex101.com/) for testing and sharing your creations.

I'll leave you with one parting bit of knowledge about regex: how to write recursive expressions.

Simple recursions are quite easy, really, but let's think about what that means in the context of a regular expression. [The syntax for a simple recursion](https://www.regular-expressions.info/) in a regular expression is `(?R)?`. But of course this syntax must appear within the expression itself. So what we're doing is _nesting_ the expression inside itself, an arbitrary number of times. For example:

```python
pattern: (hey(?R)?oh)
string:  heyoh heyyoh heyheyohoh hey oh heyhey heyheyheyohoh
matches: ^^^^^        ^^^^^^^^^^                  ^^^^^^^^^^
groups:  11111        1111111111                  1111111111
```

([See this in action here.](https://regex101.com/r/Mf9L3O/110))

Since the nested expression is optional (`(?R)` is followed by a `?`), the simplest match is to just ignore the recursion completely. So `hey` followed by `oh` (`heyoh`) matches. To match any expression more complex than that, we must find this matched substring nested inside itself at the point in the expression which we inserted the `(?R)` sequence. In other words, we could find `heyheyohoh` or `heyheyheyohohoh`, and so on.

One cool thing about these nested expressions is that, unlike backreferences and named capturing groups, they don't limit you to matching the exact text that you matched previously, character-for-character. For instance:

```python
pattern: ([Hh][Ee][Yy](?R)?oh)
string:  heyoh heyyoh hEyHeYohoh hey oh heyhey hEyHeYHEyohohoh
matches: ^^^^^        ^^^^^^^^^^               ^^^^^^^^^^^^^^^
groups:  11111        1111111111               111111111111111
```

([See this in action here.](https://regex101.com/r/Mf9L3O/111))

You can imagine that the regex engine is literally copying and pasting your regular expression inside of itself an arbitrary number of times. Of course, this means that sometimes it might not do what you might have hoped:

```python
pattern: ((?:\(\*)[^*)]*(?R)?(?:\*\)))
string:  (* comment (* nested *) not *)
matches:            ^^^^^^^^^^^^
groups:             111111111111
```

([See this in action here.](https://regex101.com/r/Mf9L3O/113))

Can you tell why that regular expression only captured the innermost nested comment, and not the outer comment as well? One thing is for sure: when writing complicated regular expressions _always test them_ to ensure they're working the way you think they are.

[[ back to Table of Contents ]](#toc)

---

I hope you enjoyed this whirlwind introduction to the beautiful world of regular expressions. If you have any questions or find any errors in my tutorial, please let me know in the comments below. Also please share this guide with anyone you know who needs a gentle introduction to the wonders of regex.

As always, thanks for reading!

---
---

Follow me on [Dev.To](https://dev.to/awwsmm) and [Twitter.com](https://twitter.com/_awwsmm) (but not in real life, please).

One day, someone will buy me a coffee on [Ko-Fi.com](https://ko-fi.com/awwsmm) and I will be very grateful. Donate if you'd like to support my work, and make it possible for me to produce more guides like this one.