Powerful regex for the practical dev #2
Tags: regex, search
Published: 2020-09-14 20:20:31

---
title: Powerful regex for the practical dev #2
description: Practical regex case study for developers
published: true
date: 2020-09-12 00:00:00 UTC
tags: regex,search
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/73a33pqtm71y8x1aczb5.jpg
canonical_url: https://kangabru.xyz/2020/09/12/practical-regex-2.html
series: Practical regex
---

As a developer you’re constantly working with large amounts of text like source code, logs, and data files. Often you need to extract, replace, or manipulate that text and regex can help you.

Today I present variants of a real case study where I personally used regex. These examples build upon my last post so check that out too.

{% link https://dev.to/kanga_bru/powerful-regex-for-the-practical-dev-4pj6 %}

* * *

## Case study example

This is a snippet of [this csv file](https://gist.github.com/kangabru/2d20fe8089223fe91c34169cd71ba5f2) which you can also [play with here](https://regexr.com/5c19f).

```
reddit_id,colorblind_comment,score,title,url,created
...
8cwcbu,False,101457,"Cause of Death - Reality vs. Google vs. Media [OC]",https://i.imgur.com/GtIzEok.gif,1523970172
8bzdr8,False,99626,"Gaze and foot placement when walking over rough terrain (article link in comments) [OC]",https://v.redd.it/h0f0m4v5nor01,1523628194
fpga3f,False,99488,"[OC] To show just how insane this week's unemployment numbers are, I animated initial unemployment insurance claims from 1967 until now. These numbers are just astonishing.",https://i.redd.it/tch0t0is32p41.gif,1585245693
i2vx78,True,98638,"The environmental impact of Beyond Meat and a beef patty [OC]",https://i.redd.it/jskjkodg3se51.png,1596456703
fxoxti,False,98067,"Coronavirus Deaths vs Other Epidemics From Day of First Death (Since 2000) [OC]",https://v.redd.it/yemjrb1p9rr41,1586422082
...
```

`(Top 100 posts from r/dataisbeautiful on Reddit where 'colorblind' is mentioned in the comments)`

* * *

## Text extraction: Extract urls from data

There are a ton of things you could extract here but I’ll go with this:

`Extract image urls from posts with colorblind comments`

You _could_ use excel or write a script here but this is why I would use regex:

- Quick and easily done from your IDE
- Not limited to CSVs - this could be source code which _isn’t_ so easy via excel/scripts
- Took me under a minute to do (with prior regex knowledge)

* * *

**Step #1:** Select all lines where the colorblind column is `True`

`.*,True,.*`

![](https://kangabru.xyz/images/posts/2020-09-12/regex/regex-01.jpg)

- `True` match the word ‘True’
- `,` match commas (so we don’t match ‘True’ in the title text)
- `.*` match everything before and after

* * *

**Step #2:** Limit to lines that have an image url

`.*,True,.*http.*\.(png|jpg).*`

![](https://kangabru.xyz/images/posts/2020-09-12/regex/regex-02.jpg)

- `http.*` match start of url (not necessary but needed for the next step)
- `\.(png|jpg)` match `.png` or `.jpg` extensions
- `http.*\.(png|jpg)` combine to match urls with an image extension
- `.*,True,.*` + `http.*\.(png|jpg)` + `.*` combine step 1 regex, the image url, and everything after. Now it matches lines with colorblind comments AND image urls

* * *

**Step #3:** Match just the url:

`(?<=.*,True,.*,)http.*\.(png|jpg)(?=.*)`

![](https://kangabru.xyz/images/posts/2020-09-12/regex/regex-03.jpg)

Lets break up our regex so far into parts:

| `part_1` | Before the url | `.*,True,.*` |
| `part_2` | The url | `http.*\.(png|jpg)` |
| `part_3` | After the url | `.*` |

We want to isolate `part_2` so must exclude `part_1` and `part_3` like this:

- `(?<= ... ) ...` exclude stuff _before_ a match
- `... (?= ... )` exclude stuff _after_ a match
- `(?<= part_1 )` + `part_2` + `(?= part_3 )` exclude `part_1` and `part_3`
- `(?<=.*,True,.*)http.*\.(png|jpg)(?=.*)` substitute in our actual regex

* * *

**Step #4:** Extract the text

![](https://kangabru.xyz/images/posts/2020-09-12/regex/regex-04.jpg)

A modern IDE should now let you select your matches. In VS Code you can select and copy your matches like this:

- `Alt + Enter` to select matches
- `Ctrl + c` to copy matches
- `Ctrl + v` to paste matches (in a separate file)

Urls successfully extracted! You could take this further by running this over your _entire_ repo using global search (`Alt + Shift + f`). Show me an excel script that does that!

* * *

## Text replace: Reformat url structure

Sometimes we don’t want to extract text but rather update it. For this example we’ll update all image urls in the file like so:

- from this: `https://i.redd.it/jskjkodg3se51.png`
- to this: `https://i.redd.it/png/jskjkodg3se51`

* * *

**Step #1:** Match image urls (with ID match)

`http.*/.*\.(png|jpg)`

![](https://kangabru.xyz/images/posts/2020-09-12/regex/regex-05.jpg)

- This the url regex we made before but with an image ID match that we’ll use in the next step
- `http.*` match start of url and everything after
- `/.*` match forward slash and everything after (matches the image ID)
- `\.(png|jpg)` match `.png` or `.jpg` extensions

* * *

**Step #2:** Create regex groups

`(http.*)/(.*)\.(png|jpg)`

![](https://kangabru.xyz/images/posts/2020-09-12/regex/regex-06.jpg)

Everything wrapped with `()` creates a regex group which we can reference in our replace command.

Given the url `https://i.redd.it/jskjkodg3se51.png` we must create groups for the following parts to transform them as desired:

- `https://i.redd.it/` the domain part
- `jskjkodg3se51` the image ID part
- `png` the image extension part

Let’s group those parts of our regex:

- `(http.*)` groups the domain part
- `/(.*)` groups the image ID part (note the slash is excluded)
- `\.(png|jpg)` groups the image extension part (note this was already grouped and that `.` is excluded)

* * *

**Step #3:** Replace

`(http.*)/(.*)\.(png|jpg)` + `$1/$3/$2`

![](https://kangabru.xyz/images/posts/2020-09-12/regex/regex-07.jpg)

![](https://kangabru.xyz/images/posts/2020-09-12/regex/regex-08.jpg)

Open the replace UI (`Ctrl + h` in most IDEs)

We can now reference groups in replace commands as follows:

- `$0` is the entire match
- `$1`, `$2`, `$3` is the first, second, third group etc

Our search and replace commands will therefore look like this:

- Search: `(http.*)/(.*)\.(png|jpg)`
- Replace: `$1/$3/$2`

That regex replace will change urls like so:

- `https://i.redd.it/jskjkodg3se51.png` -> `https://i.redd.it/png/jskjkodg3se51`
- `https://i.redd.it/mkdikcce8gh31.jpg` -> `https://i.redd.it/jpg/mkdikcce8gh31`

Nice huh? You can also do this globally via `Alt + shift + h` to search and replace over your whole repo. Very useful.

* * *

Powerful stuff eh? Once you learn how to build up a regex like this (hint: practice makes perfect) you can do this kind of stuff in seconds. Searching, extracting, and transforming text will become second nature.

Well that’s it for now! I’ve got 2 more sections of this case study ready to go but this article was getting long. So next week we’ll learn how to use multi-cursors to augment regex even further.

It’s seriously powerful stuff! 

Found this useful? [Tweet about it](https://twitter.com/intent/tweet?url=https%3A%2F%2Fdev.to%2Fkanga_bru%2Fpowerful-regex-for-the-practical-dev-2-i94&via=kanga_bru&text=Wow%20regex%20is%20powerful%21%20If%20you%27re%20a%20dev%20and%20NOT%20using%20regex%20then%20you%27re%20missing%20out.&hashtags=regex) or [follow me](https://twitter.com/kanga_bru) on Twitter. Until next time 👋