Airflow, Python and String Concatenation
Tags: python, airflow, strings
Published: 2020-10-26 14:51:37

This is the second part of the series about "Python concepts for people who are in need of using Apache Airflow but have little or no knowledge of Python".

While Airflow concepts will be explained [en passant](https://en.wikipedia.org/wiki/En_passant), the main focus of these articles are Python concepts and techniques.

In this article I will focus on string concatenation or to put together a text using multiple pieces like hard coded strings, variables, and/or templates.

# Why string concatenation?
Initially I wanted to dedicate this article to Python data structures (like lists and dictionaries), but to show them in practice most of my Airflow related examples where about strings.

Feel free to skip this post, or quickly skim it, if you are familiar with Python strings.

Other reasons why I wanted to focus a bit more on this topic:
- over time Python introduced multiple ways to do string concatenation, looking at code written by other people can be sometimes confusing
- it is a topic for interview questions, especially for juniors engineers or analysts
- it is easy to do it wrong (wrong = in a way which is hard to maintain)

Some people find string concatenation confusing, it can be. I hope at the end of this article you can have the tools to understand string concatenation and formatting. If not, feel free to write your feedback in the comments.

# A basic dag with strings
```python
from datetime import datetime

from airflow.models import DAG
from airflow.operators.dummy_operator import DummyOperator

dag_name = "string_sample"
task1_name = "start"
task2_name = "end"

my_dag = DAG(dag_id=dag_name + "_dag",
             start_date=datetime(2020, 4, 29),
             schedule_interval="0 0 * * *"
             )

task1 = DummyOperator(dag=my_dag,
                      task_id="task_{}".format(task1_name)
                      )

task2 = DummyOperator(dag=my_dag,
                      task_id=f"task_{task2_name}"
                      )

task1 >> task2
```

If we look into the Airflow web UI we will see a DAG called `string_sample_dag`, which consists of two tasks: `task_start` and `task_end`. The operators used are, again, the DummyOperator ones. This DAG is just an excuse to show you how to concatenate and format strings in Python.

# The many ways to concatenate strings in Python

## The lazy way, the `+`
The simplest way to concatenate multiple strings is to use the plus sign, `+`:
```python
dag_name + "_dag"
```
It works well adding strings, here another working example:
```python
>>> text = "world"
>>> print("hello " + text)
hello world
```
And here is when being lazy doesn't work that well:
```python
>>> text = 123
>>> print("hello " + text)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
TypeError: must be str, not int
```
The `+` operator is a simple man, it sees values it tries to concatenate them. Sometimes it works, sometimes it doesn't. In our case, it is not able to join a string with an integer. To make it work we need to transform `123` in a string `"123"`:
```python
>>> print("hello " + str(text))
hello 123
```

## Better formatting with `.format()`
In the first task of the DAG there is a second, better way to concatenate strings. The string method `.format()` allows us to format a string that we use as template:

```python
"task_{}".format(task1_name)
```
In our previous "hello world" example the template for our greeting is `"hello {}"`, where instead of `{}` we want to have `world`, `123` or maybe `John`.

```python
>>> text = "world"
>>> print("hello {}".format(text))
hello world
```

And it works with numbers too:

```python
>>> text = 123
>>> print("hello {}".format(text))
hello 123
```

### With multiple values
```python
>>> greeting = "hello"
>>> text = "world"
>>> print("{} {}".format(text, greeting))
world hello
```
Uhm, this doesn't look right. The `.format()` method assigns the values to the `{}` placeholders in the order they appear. Of course we could switch the order of the passed values, but what if over time we change the template?

To avoid this use ordinal numbers or, even better, name the placeholders. 

First with numbers:
```python
print("{1} {0}".format(text, greeting))
hello world
```
**Note** The first value has index 0, the second 1.

Naming the placeholders:
```python
>>> print("{hi} {who}".format(who=text, hi=greeting))
hello world
```

Using names placeholders our template string is more meaningful (which will be nice reviewing this code after a while), but we need to specify which value is `hi` and which is `who`.

For more formatting options and examples you can take a look at the Python [documentation here](https://docs.python.org/3/library/string.html#format-string-syntax). You probably will not need them for your DAGs, but it is good to know where to start (beside using Stack Overflow).

## The `f` string

Python 3.6 introduced a new way to format strings:
```python
>>> text = "world"
>>> print(f"hello {text}")
hello world
```
The `f` stands for formatted string. But also for fast, because this formatting method is faster to write and faster to produce the final result.

In a formatted string the values in between `{}` are expressions that will be evaluated at runtime. Therefore it is possible to do things like:

```python
>>> print(f"{5 - 4} hello to {text.lower()}")
1 hello to world
```

Again, more use case and advanced option can be discovered reading the Python [documentation](https://docs.python.org/3/reference/lexical_analysis.html#f-strings).

# Few additional things on Python strings
These are probably details that you will not use writing DAGs, but it is worth to mention to complete the basic overview on the strings.

You probably noticed that formatting a string in Python utilizes the curly braces, then how to format a string that contains curly braces? Double curly, duh:
```python
>>> print(f"In {{text}} is {text}")
In {text} is world
```

What about the double quote sign `"`? First, in Python you can use `'` and `"` for a string, and you need to use the same sign to close the string. Another why is to escape the quote sign using a `\`:

```python
>>> print('This is a double quote: "')
This is a double quote: "
>>> print("This is a double quote too: \"")
This is a double quote too: "
```

Finally, how to deal with very long strings. Python allows you to break a long line of code into multiple using the backslash `\`:
```python
>>> print("hello " \
... "world")
hello world
```
As you can see Python concatenates the two strings ignoring the new line (to achieve that you can do `"hello \n"`).

But you can also create a multi line string using `"""` or `'''`:
```python
>>> print("""hello
>>> world""")
hello
world
```

## More additional things
Actually there is much more to say about Python strings,  concatenation, and formatting, but this will require another post (or more).

Personally I believe these things are not very interesting for someone approaching Python for the first time. But I have been wrong in the past, vote unicorn to tell me that I am wrong and you want to know more about strings (and maybe add a comment on what you find difficult with strings in Python). 

# Which one to use?
![Use f-strings](https://dev-to-uploads.s3.amazonaws.com/i/1wufbtsbz7pyhg6a2hlm.png)
Using Python 3.6 or above, `f`-string is the way to go: it is fast to write and easy to maintain.

But I am the first to admit that for quick debugging I still resort to the `+` (when possible).

The `.format()` is worth to mention because there are many examples in the wild using it or you can encounter some old code which requires your attention.

That said, I suggest you to take a look at the links to the documentation to be aware of the formatting possibility offered by Python, sooner or later you will need them.

# Shameless plug
In case you need support or assistance feel free to reach out to me in the comment or direct messages. On twitter you can find me with the handler @mucio.

If you need more structured help, the nice people at [Untitled Data Company](https://untitleddata.company/) (which includes me) will be happy to help you with all your data needs.

### Credits
Cover photo by [Jess Bailey](https://unsplash.com/@jessbaileydesigns?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText) on [Unsplash](https://unsplash.com/s/photos/tape-craft-scissors?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText).