Why is my regex working intermittently?
Tags: debugging, javascript, beginners, regex
Published: 2020-01-13 07:36:56

---
title: Why is my regex working intermittently?
published: true
description: Why is my Regular Expression failing every other time it is called? Nasty weird bugs caused by silly things
cover_image: https://thepracticaldev.s3.amazonaws.com/i/ft4sgg6n1xtm2bg71nyq.png
tags: #debugging, #javascript, #beginners, #regularexpressions
---

![wtf + facepalm](https://media.giphy.com/media/SEp6Zq6ZkzUNW/giphy.gif)
also 
#### Why is my Regular Expression failing every other time it is called?
#### Why is my RegEx working only the first time but not the second ( and working again the third?)
#### Nasty weird bugs caused by silly things

---
## TLDR
When the _RegExp test method_ is run with **global flag** (*/g*), the regex keeps internally the state of the search. Therefore at each invocation the regular exception will be run from the last index that was found previously.
```
const regex = /a/gi  // const regex = RegExp('a', 'gi')
regex.test("abc")  // --> true
regex.test("abc")  // --> false
regex.test("abc")  // --> true
```

## Solution
- If not strictly necessary, avoid the global flag or
- Use _String.match(RegExp)_ instead of _RegExp.test(String)_ or
```
"abc".match(regex) // --> ["a"]
"abc".match(regex) // --> ["a"]
"abc".match(regex) // --> ["a"]
```
- Recreate the Regex at each invocation (avoid a reference to a constant or any regex define elsewhere)

```
/a/gi.test("abc")  // --> true
/a/gi.test("abc")  // --> true
/a/gi.test("abc")  // --> true
```
---

When we want to know whether a pattern is found in a string we can most commonly use two approaches:

we can check if the string matches with the regex (`myString.match(myRegex)`)
or 
we can test the regex against the string (`myRegex.test(myString)`)

If I am not interested in finding the matches, and I just want to know if we found something or not, I prefer to use RegExp.test which is simpler returns a boolean instead of an array ( and it´s also slightly faster).

The other day I noticed a weird behavior on one of a lambda we recently deployed to staging. 

In that Lambda we have a [https://github.com/sindresorhus/p-waterfall](pipeline of async operations) :


- parse and validate User Input
- load data from DynamoDB
- load configuration from AWS ParameterStore (SSM)
- manipulate the data from dynamoDB together with the user input
- compose a URL using the loaded configuration and the user data and validate it
- fetch data from that URL
- save result to a AWS SQS

We are still in the MVP stage and we have _some_ unit tests and integration tests in place.
Everything was working fine in the tests and even after deployment. But we noticed that the behavior when deployed was a bit weird.  the lambda was returning an error, every now and then, intermittently with apparently no reason, since the payload was always the same.

After activating some logging I realized that the composed URL was invalid, therefore I started looking at the configuration and data being loaded from DynamoDB or SSM - maybe some Permissions/Policies missing? (remember that when running locally with serverless offline the credentials and permissions are your own - and therefore different from those in the lambda container).
After some investigation, I found out that the composed URL was always the same and what was failing was the url-validation method - even though the input URL was exactly the same...

I could immediately recall some behaviour in the Regular Expression related to a shifting index during the search therefore I opened the _RegExp.test_ [Docs](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp/test) Gotcha!


### Using test() on a regex with the global flag

> If the regex has the global flag set, test() will advance the lastIndex of the regex. **A subsequent use of test() will start the search at the substring of str specified by lastIndex**. It is worth noting that the lastIndex will not reset when testing a different string.

### What does it mean exactly? 

It means for example that the regex `/a/gi/` will test the string "abc" and find a match at the first character.  When the regex is run again, it will start testing from that point on, therefore on "bc". Since on "bc" the regex can't find a match, it will restart from zero the next time, and here you go "abc" matches again.  
If you think that this will happen even if you try another string like "axy" or that with a string containing more than one _a_ the rhythm of successful matches becomes irregular, such behaviour could lead to quite nasty bugs.

```
const regex = /a/gi  // const regex = RegExp('a', 'gi')
regex2.test("abcad")  // --> true
regex2.test("abcad")  // --> true
regex2.test("abcad")  // --> false
regex2.test("abcad")  // --> true
```
![banging head on desk](https://media.giphy.com/media/xTiTnslZ0E5sqMbEac/giphy.gif)

As I said, this is documented pretty well and while writing this I tried out some other references and found of course [lots of]( https://stackoverflow.com/questions/13233654/javascript-regular-expression-is-working-fine-first-time-but-not-second-time-wo) similar [questions](https://stackoverflow.com/questions/3811890/javascript-regular-expression-fails-every-other-time-it-is-called) on [StackOverflow](https://stackoverflow.com/questions/1520800/why-does-a-regexp-with-global-flag-give-wrong-results) - as old as 10 years! -  so it should not have been a surprise, but it indeed caught us off guard.

I adjusted my method - wrote a couple more unit tests to check this edge case and that was it.
![awesome](https://media.giphy.com/media/d2Z9QYzA2aidiWn6/giphy.gif) 


For some more fun/insights about RegEx checkout [https://blog.codinghorror.com/regular-expressions-now-you-have-two-problems/](this post from CodingHorror)

Hope it helps.

---
Photo by [Mr Cup / Fabien Barral](https://unsplash.com/@iammrcup?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on Unsplash