Python: How simple string concatenation can kill your code performance
Tags: python, tutorial, beginners, performance
Published: 2023-03-18 13:25:26


> Note: a video version of this tutorial is available  [Python: How simple string concatenation can kill your code performance](https://youtu.be/P0iaH1a0TQM)
> 

---

### In general, there is no problem with string concatenation

String concatenation is one of the first and most used features in programming. You learn variables basics, integers, booleans, and strings. It’s even one of the first things we learn in the python official Documentation:


![String concatenation from Python offical documentation](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/379t97cs2ct174wwedl7.png)

So yes, string concatenation is not bad, and it might even be impossible to do programming without needing to concatenate strings for one reason or another.

Although, yes, like almost any useful tool or feature, when used the wrong way it can lead to really bad effects. So it’s important to be aware of how to use them correctly.

### When string concatenation could be dangerous?

You should pay attention to string concatenation when those two conditions are met:

1. The feature is a performance-critical one. (Example: a text data analysis feature on a web application that is running, and a user in the website interface is waiting to get back the result, so your code should be as fast as possible)
2.  You’re dealing with a lot of concatenation (starting from 10000) because of a loop maybe, and/or the strings you’re concatenating are really really long.

That can lead to significant overhead in terms of memory allocation and garbage collection, slow down your code, and in the worse case just make your code crash in production :grimacing:.

And, that also means it's totally safe to use string concatenation when those conditions are not met, and you’re sure that, these conditions can never be met in the future. So for instance:

```python
firstname = "John"
lastname = "Joe"
fullname = fistname + lastname
```

is totally okay because there is no way someone’s name can become long enough to create a performance issue. Anyways, not in this world :sweat_smile:. 

### Why string concatenation is dangerous in those cases?

In Python, strings are immutable objects, which means that every time a string is concatenated with another string, a new string object is created in memory. 

To illustrate that let's run this simple code:

```python
my_str = "Hello"
print(f"my_str: {my_str}")
print(f"id = {id(my_str)}")
my_str += " World"
print(f"\n my_str: {my_str}")
print(f"id = {id(my_str)}")
```

> Note: Python `id()` function returns the `identity` of an object. The identity of an object is **an integer, which is guaranteed to be unique and constant for this object during its lifetime**.
> 

The output will be:

```python
my_str: Hello
id = 140575060484848

my_str: Hello World
id = 140575060532336
```

As you can see the `id` of the string before the concatenation and after are not the same so python has created a new string.

### The solution to that issue and a comparison with string concatenation

To address this performance issue, you can use alternative techniques such as using a list to hold the individual string components and then joining them together using the **`join`** method. 

Python doesn’t create a new list each time you append an element, and the join method is optimized to avoid creating a new string object for each string in the list it has to join.

```python
import timeit

# Concatenate strings
start_time = timeit.default_timer()
s = ''
for i in range(10000):
    s += 'a'
elapsed_time_concat = timeit.default_timer() - start_time
print('Time taken for string concatenation:', elapsed_time_concat)

# Join strings
start_time = timeit.default_timer()
lst = []
for i in range(10000):
    lst.append('a')
s = ''.join(lst)
elapsed_time_join = timeit.default_timer() - start_time
print('Time taken for string join:         ', elapsed_time_join)

join_performance = (elapsed_time_concat - elapsed_time_join) / elapsed_time_concat
print('Performance gained with join', join_performance)
```

In this example, we generate a string of length 10,000 by concatenating the string `a` 10,000 times using both the string concatenation and the join method. The output shows that the join method is significantly faster than the concatenation method:

```
Time taken for string concatenation: 0.0019936899916501716
Time taken for string join:          0.0011655330017674714
Performance gained with join 41.53890491255549 %
```

As you can see, by using `join` we have increased the time performance by almost  `42%` which is wonderful.

### Diving deep into the why: Math and a Big(O) explanation

This behavior could also be explained using BigO notation especially, the **time complexity**.

Let’s focus on the concatenation part of our previous code.

We have a string `s`  that we concatenate `M` times with a string `c` of length on length `N`.

```python
s = ''
c = 'c'
for i in range(10000):
    s += c
```

 So here `N=1`  and `M = 10000`    

Keep this in mind:

> The complexity to create a string of length `N` is O(N).
So because concatenation creates a new string
of length `len(s+c)` , the complexity of the concatenation at each iteration is the `length of s at the previous iteration` + `length of string c`.
> 

![Table: Illustrate string concatenation time complexity](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7xxoxgys9s8are9tomu5.png)
As you can see the total, or overall complexity is `1 + 2 + 3 + 4 + 5 +…+ M` .
which we can mathematically approximate at `(M(M+1)) / 2`   .
And by neglecting the minority exponents we have:


![String concatenation time complexity Big O](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/doh4mlgxnkom82jrslla.png)

Let’s try to so see the time complexity with the `join` approach:

```python
lst = []
c = 'c'
for i in range(10000):
    lst.append(c)
s = ''.join(lst)
```

In this case, we have two steps:

1. The loop to create the list

```python
for i in range(10000):
    lst.append(c)
```

The complexity here is O(M), and in our case `M=1000`  because python has to execute `lst.append(c)` `M` times.

2. The join `s = ''.join(lst)`
Here the complexity is O(T) where `T = total length of the final string after the join` and in our case `T = N * M` . 

So the final time complexity is `NM + M = M (N + 1)` and we can approximate it as `O(MN)`.

Which is far better than `O(M**2)`  for the simple concatenation.

---

That is for this tutorial. I hope you understand when and why string concatenation could be dangerous so you can use `join` instead when it’s needed.

Thanks for reading, don’t forget to checkout the video version on our youtube channel [here](https://youtu.be/P0iaH1a0TQM) and I see you at the next one.

Take care.