Dead Simple Python: Working with Files
Tags: python, beginners
Published: 2019-12-21 03:14:50

---
title: Dead Simple Python: Working with Files
published: true
description: Working with text files in Python sometimes feels like it has some unexpected mysteries. Let's solve a few.
tags: python, beginners
cover_image: https://thepracticaldev.s3.amazonaws.com/i/o11gn43e9u2h1g7z3qzf.jpg
series: Dead Simple Python
---

Like the articles? Buy the book! [***Dead Simple Python*** by Jason C. McDonald is available from No Starch Press.](https://nostarch.com/dead-simple-python)

---

For me, any project doesn't start feeling "real" until the program starts reading from or writing to external files.

Unfortunately, this is one of those topics that often suffers the most from tutorial oversimplification. If only someone would document all the weirdness with files in one place...

# Basic File Opening

Let's start with the most basic file reading example. Assume I have a file in the same directory as my code file, entitled `journal1.txt`.

The standard way of opening files is using the built-in `open()` function, which is imported by default from the `io` module.

```python
file = open("journal1.txt", 'r')
for line in file:
    print(line)
file.close()
```

The `open()` function accepts a number of arguments for interacting with files in advanced ways, but most of the time, you'll only need the first two.

The first argument, `file`, accepts a string containing either an absolute or relative path to the file being opened. This is the only strictly required argument.

The second argument, `mode`, accepts a string indicating the *file mode*. If this is not specified, `'rt'` will be used, meaning it will read the file as text. The mode `'r'` is effectively the same thing, since text-mode (`t`) is part of the default behavior.

I could have just used this line instead and gotten the same behavior...

```python
file = open("journal1.txt")
```

...but I personally prefer to explicitly indicate whether I'm reading (`r`), writing (`w`), or what have you.

**All "file" objects returned from `open()` are iterable.** In the case of text files, a `TextIOWrapper` object is returned. In my example above, I iterate over the lines in the `TextIOWrapper` object `file`, and I print out each line.

Once I'm done working with the file, I need to close it with `file.close()`. It's important *not* to rely on the garbage collector to close files for you, as such behavior is neither guaranteed nor portable across implementations. Additionally, Python isn't guaranteed to finish writing to a file until `.close()` is called.

Running that code, at least in my case, prints out the contents of `journal1.txt`:

```
Could this be where all missing things are found?

Magic Hair Only for the Pure of Heart

Not naturally occurring?

Could result in minor gravity anomalies!
```

## Context Managers

In practice, always remembering to call `close()` can be a royal pain, especially once you factor in possible errors with opening a file. Thankfully, there's an even cleaner way: **context managers**!

A **context manager** is defined by the `with` statement in Python. I can rewrite my earlier code using this syntax:

```python
with open("journal1.txt", 'r') as file:
    for line in file:
        print(line)
```

The `open()` function is called, and if it succeeds, the resulting `TextIOWrapper` object is stored in `file`, and can be used in the body of the `with` statement. `file.close()` is implicitly called once control leaves the `with` statement; you never have to remember to call it!

We'll cover this in more depth in the next chapter.

# File Modes

The documentation mentions several modes which can be used with `open()`:

* `r` opens the file for reading (the default).
* `w` opens or creates the file for writing, deleting (truncating) its contents first.
* `a` opens or creates the file for writing, but appends to the end instead of truncating.
* `x` creates and opens a new file for writing; it cannot open existing files.
* `+` opens the file for both read and write (see table below).
* `t` works with the file in text mode (the default).
* `b` works with the file in binary mode.

These mode flags can be combined together. For example, `a+b` allowing writing and reading, where writing appends to the end of the file, in binary mode.

The `+` flag is always combined with another flag. When combined with `r`, it adds the functionality of `a`, except it starts at the beginning of the file (without truncating). When combined with `w`, `a`, or `x`, it allows reading as well.

The behavior of the different flags is best understood with this table, adapted from [this Stack Overflow answer by industryworker3595112](https://stackoverflow.com/a/30931305/472647):

```
                     | r   r+   w   w+   a   a+   x   x+
---------------------|----------------------------------
allow read           | ✓   ✓        ✓        ✓        ✓
allow write          |     ✓    ✓   ✓    ✓   ✓    ✓   ✓
create new file      |          ✓   ✓    ✓   ✓    ✓   ✓
open existing file   | ✓   ✓    ✓   ✓    ✓   ✓
erase file contents  |          ✓   ✓
allow seek           |     ✓    ✓   ✓             ✓   ✓
position at start    | ✓   ✓    ✓   ✓             ✓   ✓
position at end      |                   ✓   ✓
```

# Reading

We can read from a file in text mode using using either the `read()`, `readline()`, or `readlines()` functions, or by iterating over it directly.

Of course, this requires that the file be opened for reading, using the appropriate file mode flags (see "File Modes" section). If you ever need to check whether an object `file` can be read, use the `file.readable()` function.

Let's contrast the three ways of reading from a file:

## read()

The `read()` function reads the entire contents of the file as one long string.

```python
with open("journal1.txt", 'r') as file:
    contents = file.read()
    print(contents)

# Could this be where all missing things are found?
# Magic Hair Only for the Pure of Heart
# Not naturally occurring?
# Could result in minor gravity anomalies!
```

Alternatively, you can tell `read()` the maximum number of characters to read from the file stream:

```python
with open("journal1.txt", 'r') as file:
    contents = file.read(20)
    print(contents)

# Could this be where
```

## readline()

The `readline()` function behaves exactly like `read()`, except it stops reading when it encounters a line break. The line break is included in the returned string.

```python
with open("journal1.txt", 'r') as file:
    contents = file.readline()
    print(contents)

# Could this be where all missing things are found?
```

As with `read()`, you can specify the maximum number of characters to be read:

```python
with open("journal1.txt", 'r') as file:
    contents = file.readline(20)
    print(contents)

# Could this be where
```

## readlines()

The `readlines()` function returns the entire file as a list of strings, with each string being one line.

```python
with open("journal1.txt", 'r') as file:
    contents = file.readlines()
    for c in contents:
        print(c)

# Could this be where all missing things are found?
#
# Magic Hair Only for the Pure of Heart
#
# Not naturally occurring?
#
# Could result in minor gravity anomalies!
#
```

You'll notice that the newline character is included with each line. We can remove that by calling the `.strip()` function on each string.


```python
with open("journal1.txt", 'r') as file:
    contents = file.readlines()
    for c in contents:
        print(c.strip())

# Could this be where all missing things are found?
# Magic Hair Only for the Pure of Heart
# Not naturally occurring?
# Could result in minor gravity anomalies!
```

You can also limit how much is read from the file by specifying a maximum number of characters. Unlike before, however, this is *not* a hard limit. Instead, once the specified limit has been exceeded by the total number of characters read from all lines so far, only the rest of the current line will be read.

This is best understood by comparing `read()` and `readlines()`. First, I'll read to a hard limit of 60 characters:

```python
with open("journal1.txt", 'r') as file:
    contents = file.read(60)
    print(contents)

# Could this be where all missing things are found?
# Magic Hair
```

Compare that to calling `readlines()` with a "hint" of 60 characters:

```python
with open("journal1.txt", 'r') as file:
    contents = file.readlines(60)
    for c in contents:
        print(c.strip())

# Could this be where all missing things are found?
# Magic Hair Only for the Pure of Heart
```

In that second example, the entirety of the first two lines is read, but no more.

Unlike with the other two functions, `readlines()` only always reads the whole line.

## Iteration

As you saw earlier, we can iterate over a file directly:

```python
with open("journal1.txt", 'r') as file:
    for line in file:
        print(line)

# Could this be where all missing things are found?
# Magic Hair Only for the Pure of Heart
# Not naturally occurring?
# Could result in minor gravity anomalies!
```

This is functionally the same as:

```python
with open("journal1.txt", 'r') as file:
    for line in file.readlines():
        print(line)
```

The difference between the two is that the first approach, the direct iteration, is *lazy*, whereas the second approach reads the whole file first, before iterating over the contents.

# Writing

We can write to a file in much the same way, using either the `write()` or `writelines()` functions.

This requires that the file be opened for writing (see "File Modes" section). The `file.writable()` function can be used to check if the `file` object is writable.

> In the examples for this section, I'll show the file contents in the comment at the bottom.

## write()

The `write()` function writes the given lines to a file.

I can write an entire multiline string to a new file called `journal3.txt` with `write()`, like so:

```python
entry = """If you go on enough road trips
chances are, you've seen a
certain bumper sticker:
WHAT IS THE MYSTERY SHACK?
"""

with open("journal3.txt", 'x') as file:
    file.write(entry)

# If you go on enough road trips
# chances are, you've seen a
# certain bumper sticker:
# WHAT IS THE MYSTERY SHACK?
#
```

As long as `journal3.txt` does *not* already exist, it will be created with the given contents.

I can override the entire contents of `journal3.txt` using the `w` file mode:

```python
with open("journal3.txt", 'w') as file:
    file.write("GNOMES\nWEAKNESS?\n")

# GNOMES
# WEAKNESS?
#
```

> **GOTCHA ALERT:** Watch your file mode! `w` and `w+` will erase the entire contents of the file. Use either `a` or `a+` to write to the end of the file.

I can append to the file instead using the `a` file mode:

```python
with open("journal3.txt", 'a') as file:
    file.write("leaf blowers\n")

# GNOMES
# WEAKNESS?
# leaf blowers
#
```

The `write()` function also returns an integer representing the number of characters written.

## writelines()

The `writelines()` function writes a list of strings to a file.

```python
lines = [
    "Finally back safe and sound\n",
    "from one of the weirdest days\n",
    "at Gravity Falls.\n"
]

with open("journal3.txt", 'w') as file:
    file.writelines(lines)

# Finally back safe and sound
# from one of the weirdest days
# at Gravity Falls.
#
```

Unlike with with `write()`, the `writelines()` function only ever returns `None`.

# Seeking

The `file.seek()` function allows you to move back and forth within a file object `file`, character-by-character. When working with text streams, it accepts one argument: a positive integer representing a new position to move to, represented as the number of characters from the beginning.

In addition to changing position, the `file.seek()` function will also return an integer representing the new absolute position in the file. You can also get the current position by calling the `file.tell()` function.

The `r+` file mode is best used in conjunction with the `seek()` function, although it can be used with any other file mode *besides* `a` and `a+`.

I'll first use the `seek()` function to read only part of the `journal1.txt` file:

```python
with open("journal1.txt", 'r') as file:
    file.seek(50)
    contents = file.read(5)
    print(contents)

# MAGIC
```

I'll write a new initial version of the `journal3.txt` file:

```python
with open("journal3.txt", 'w') as file:
    file.write("FLOATING EYEBALLS")

# FLOATING EYEBALLS
```

I can use the `r+` mode to change part of this file.

> **GOTCHA ALERT:** The `write()` command will always *write over* existing contents of the file, unless you append to the end. To insert text into a file non-destructively, it's usually best to read the entire contents in as a string (or list), edit the string, and then write it back out.

Here, I'll replace the word "EYEBALLS" with "NONSENSE!":


```python
with open("journal3.txt", 'r+') as file:
    file.seek(9)
    file.write("NONSENSE!")

# FLOATING NONSENSE!
```

After opening the file, I move to the 9th character from the beginning, and then `write()` the new data.

## Seek with Binary

When you open a file in binary mode (`b`), you can move in the file in a more dynamic way, with *two* arguments instead of one:

* `offset`: the distance in characters to move (can be negative)
* `whence`: the position from which to calculate the offset: `0` for the start of the file (default), `1` for the current position, or `2` for the end of the file.

Unfortunately, use of the `whence` argument doesn't work with files opened in text mode.

# File Errors

The four most common errors relating to working with files are as follows:

## FileNotFoundError

The `r` and `r+` modes require that a file exist before opening it. Otherwise, a `FileNotFoundError` will be raised:

```python
try:
    with open("notreal.txt", 'r') as file:
        print(file.read())
except FileNotFoundError as e:
    print(e)
```

## FileExistsError

The `x` and `x+` file modes are specifically for creating a new file. If the file already exists, a `FileExistsError` will be raised:

```python
try:
    with open("journal3.txt", 'x+') as file:
        print(file.read())
except FileExistsError as e:
    print(e)
```

## UnsupportedOperation

The `io.UnsupportedOperation` error is raised whenever you try to read on a file only opened for writing, or write on a file opened only for reading:

```python
import io

try:
    with open("journal3.txt", 'w') as file:
        print(file.read())
except io.UnsupportedOperation as e:
    print(e)


try:
    with open("journal3.txt", 'r') as file:
        file.write('')
except io.UnsupportedOperation as e:
    print(e)
```

# An Issue of Line Separators

Some savvy readers will recall that, while UNIX uses `\n` as a line separator, Windows uses `\r\n`. Surely this matters when we're reading and writing files, right?

In fact, Python abstracts this out for us behind-the-scenes. **Always use `\n` as a line separator when writing files in text mode, regardless of operating system!**

# File Paths

Up to this point, I've just been using a file in the same folder as the code, but this is very rarely what we want! We need to be able to build paths to files.

The trouble is, file paths are not the same on all systems. UNIX-style systems, such as macOS and Linux, use the UNIX file path conventions, while Windows uses an entirely different scheme. Our solution has to work for *both*, meaning that harded paths are not an option.

To address this, Python offers two modules: `os` and `pathlib`.

## Creating a Path

Python actually offers multiple classes for building paths, depending on your particular needs. In most cases, however, you should just use `pathlib.Path`.

Let's say I wanted to make a special directory in the current user's home folder called `.dead_simple_python`, and then write a file to that location. Here's how I'd do that:

First, I create a `Path()` object pointing to just the final desired directory (not yet the file).

In the `Path()` constructor, I pass each part of the path as a separate string. I can use the class method `Path.home()` to acquire the path to the user directory.


```python
from pathlib import Path
import os

file_path = Path(Path.home(), ".dead_simple_python")
```

Next, I'll check if the path already exists using `file_path.exists()`, and if it doesn't exist, I'll use the `os.makedirs` function to create any of the missing directories in the path:

```python
if not file_path.exists():
    os.makedirs(file_path)
```

Finally, I can add the filename to the path object I already have, and then open that file for writing:

```python
file_path = file_path.joinpath("journal4.txt")

with file_path.open('w') as file:
    lines = [
        "If you've ever taken a road trip \n",
        "through the Pacific Northwest, you've \n",
        "probably seen a bumper sticker for a \n",
        "place called Gravity Falls.\n"
    ]
    file.writelines(lines)
```

You'll notice that I used `file_path.open('w')` instead of `open(file_path, 'w')`. Technically, both do the exact same thing, although the member function is preferred.

## Relative Paths

The reason that `open("journal1.txt")` works is because it's a relative path, starting in the directory the code is being executed from.

If I have a `journals/` directory in the same directory as my code, I can use this:

```python
from pathlib import Path

file_path = Path("journals", "journal1.txt")

with file_path.open('r') as file:
    print(file.read())
```

As long as I don't start with an absolute path, such as that produced by `Path.home()`, Paths are relative.

But what if I want to move *up* a directory, instead of down? You might be tempted to use the `..`, but as you may guess, this is not guaranteed to be portable across all operating systems. Instead, I can use `os.pardir` to move to the previous directory.

Imagine we have a directory structure that looks like this:

```
example
├── code
│   └── read_file.py
└── journals
    └── journal1.txt
```

If, from within `path_relative2/code`, I run `python read_file.py`, I can access `journal1.txt` with the following:

```python
from pathlib import Path
import os

file_path = Path(os.pardir, "journals", "journal1.txt")

with file_path.open('r') as file:
    print(file.read())
```

# Review

We've only just brushed the surface of dealing with files, but hopefully this has demystified the `open()` function and `Path` objects. Here's a quick recap:

* The `open()` function has several modes: `r` for read, `w` for truncate and write, `x` for create and write, and `a` for append. Adding `+` to any of those adds the missing functionality, either reading or writing.

* You must remember to close any file you open. You can do so manually with `open()`, or...

* If you use the `with` statement (context manager) to open a file, the file will be closed automatically.

* `myfile.seek()` lets you change position in the open file `myfile`. This doesn't work with the modes `r`, `a`, or `a+`.

* A `pathlib.Path` object lets you create a portable path, build from the strings passed to the `Path()` initializer.

* You can call `open()` directly on a `Path` object.

* Use `Path.home()` to get the absolute path to the current user's home folder.

* Use `os.pardir` to access the parent directory (equivalent to `..` on Unix.)

In the next section, I'll dive into context managers in more depth.

Here's the documentation, as usual:

* [Python Reference: Built-in Functions — open()](https://docs.python.org/3/library/functions.html#open)
* [Python Reference: Compound statements — The with Statement](https://docs.python.org/3.8/reference/compound_stmts.html#the-with-statement)
* [PEP 343: The "with" statement](https://www.python.org/dev/peps/pep-0343/)
* [Python Standard Library: io — TextIOWrapper](https://docs.python.org/3/library/io.html#io.TextIOWrapper)
* [Python Standard Library: pathlib — Path](https://docs.python.org/3/library/pathlib.html#pathlib.Path)