Executing external commands from Python
Tags: python, beginners, ebook, exercises
Published: 2021-02-03 13:29:14

---
title: "Executing external commands from Python"
published: true
description: "Execute external commands and capture their output from Python"
canonical_url: https://learnbyexample.github.io/100_page_python_intro/executing-external-commands.html
series: 100 Page Python Intro
tags: Python,beginner,ebook,exercises
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/x0gbxspe60srzezgp6gd.png
---

This chapter will discuss how you can execute external commands from Python, capture their output and other relevant details such as the exit status. The availability of commands depends on the OS you are using (mine is Linux). 

## Using os module

Last chapter showed a few examples with `os` module for file processing. The `os` module is a feature rich module with lot of other uses, like providing an interface for working with external commands. Here's an example:

```python
>>> import os

>>> os.system('echo hello "$USER"')
hello learnbyexample
0
```

Similar to the `print()` function, the output of the external command, if any, is displayed on the screen. The return value is the exit status of the command, which gets displayed by default on the REPL. `0` means the command executed successfully, any other value indicates some kind of failure. As per [docs.python: os.system](https://docs.python.org/3/library/os.html#os.system):

>On Unix, the return value is the exit status of the process encoded in the format specified for `wait()`.

Here's an example for non-zero exit status:

```python
>>> status = os.system('ls xyz.txt')
ls: cannot access 'xyz.txt': No such file or directory
>>> status
512
# to get the actual exit value
>>> os.waitstatus_to_exitcode(status)
2

# if you don't want to see the error message,
# you can redirect the stderr stream
>>> os.system('ls xyz.txt 2> /dev/null')
512
```

You can use the `os.popen()` method to save the results of an external command. It provides a file object like interface for both read (default) and write. To check the status, call `close()` method on the filehandle (`None` means success).

```python
>>> fh = os.popen('wc -w <ip.txt')
>>> op = fh.read()
>>> op
'9\n'
>>> status = fh.close()
>>> print(status)
None

# if you just want the output
>>> os.popen('wc -w <ip.txt').read()
'9\n'
```

## subprocess.run

The `subprocess` module provides a more flexible and secure option to execute external commands, at the cost of being more verbose.

Quoting relevant parts from [doc.python: subprocess module](https://docs.python.org/3/library/subprocess.html):

>The `subprocess` module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes.

>The recommended approach to invoking `subprocesses` is to use the `run()` function for all use cases it can handle. For more advanced use cases, the underlying `Popen` interface can be used directly.

```python
>>> import subprocess

>>> subprocess.run('pwd')
'/home/learnbyexample/Python/programs/'
CompletedProcess(args='pwd', returncode=0)

>>> process = subprocess.run(('ls', 'xyz.txt'))
ls: cannot access 'xyz.txt': No such file or directory
>>> process.returncode
2
```

The first argument to `run()` method is the command to be executed. This can be either be a single string or a sequence of strings (if you need to pass arguments to the command being executed). By default, command output is displayed on the screen. Return value is a `CompletedProcess` object, which has relevant information for the command that was executed such as the exit status. 

As an exercise, read [subprocess.run documentation](https://docs.python.org/3/library/subprocess.html#subprocess.run) and modify the above `ls` example to:

* redirect the `stderr` stream to `/dev/null`
* automatically raise an exception when the exit status is non-zero

See also:

* [stackoverflow: How to execute a program or call a system command from Python?](https://stackoverflow.com/questions/89228/how-to-execute-a-program-or-call-a-system-command-from-python)
* [stackoverflow: difference between subprocess and os.system](https://stackoverflow.com/questions/4813238/difference-between-subprocess-popen-and-os-system)
* [stackoverflow: How to use subprocess command with pipes](https://stackoverflow.com/questions/13332268/how-to-use-subprocess-command-with-pipes)
* [stackoverflow: subprocess FAQ](https://stackoverflow.com/questions/tagged/subprocess%2bpython?tab=Votes)

## shell=True

You can also construct a single string command, similar to `os.system()`, if you set `shell` keyword argument to `True`. While this is convenient, use it only if you have total control over the command being executed such as your personal scripts. Otherwise, it can lead to security issues, see [stackoverflow: why not use shell=True](https://stackoverflow.com/questions/13491392/why-not-just-use-shell-true-in-subprocess-popen-in-python) for details.

Quoting from [docs.python: subprocess Frequently Used Arguments](https://docs.python.org/3/library/subprocess.html#frequently-used-arguments):

>If `shell` is `True`, the specified command will be executed through the shell. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of `~` to a user's home directory

```python
>>> p = subprocess.run(('echo', '$HOME'))
$HOME
>>> p = subprocess.run('echo $HOME', shell=True)
/home/learnbyexample

>>> p = subprocess.run(('ls', '*.txt'))
ls: cannot access '*.txt': No such file or directory
>>> p = subprocess.run('ls *.txt', shell=True)
ip.txt

>>> p = subprocess.run('seq -s, 10 > out.txt', shell=True)
>>> p = subprocess.run('cat out.txt', shell=True)
1,2,3,4,5,6,7,8,9,10
```

If `shell=True` cannot be used but shell features as mentioned above is needed, you can use modules like `os`, `glob`, `shutil` and so on as applicable. See also [docs.python: Replacing Older Functions with the subprocess Module](https://docs.python.org/3/library/subprocess.html#replacing-older-functions-with-the-subprocess-module).

```python
>>> p = subprocess.run(('echo', os.getenv('HOME')))
/home/learnbyexample
```

## Changing shell

By default, `/bin/sh` is the shell used for POSIX systems. You can change that by setting the `executable` argument to the shell of your choice.

```python
>>> p = subprocess.run('diff <(seq 3) <(seq 4)', shell=True)
/bin/sh: 1: Syntax error: "(" unexpected

>>> p = subprocess.run('diff <(seq 3) <(seq 4)', shell=True,
                       executable='/bin/bash')
3a4
> 4
```

## Capture output

If you use `capture_output=True`, the `CompletedProcess` object will provide `stdout` and `stderr` results as well. These are provided as `bytes` data type by default. You can change that by setting `text=True`.

```python
>>> p = subprocess.run(('date', '-u', '+%A'), capture_output=True, text=True)
>>> p
CompletedProcess(args=('date', '-u', '+%A'), returncode=0,
                 stdout='Monday\n', stderr='')
>>> p.stdout
'Monday\n'
```

You can also use `subprocess.check_output()` method to directly get the output.

```python
>>> subprocess.check_output(('date', '-u', '+%A'), text=True)
'Monday\n'
```

>You can also use legacy methods `subprocess.getstatusoutput()` and `subprocess.getoutput()` but they lack in features and do not provide secure options. See [docs.python: subprocess Legacy Shell Invocation Functions](https://docs.python.org/3/library/subprocess.html#legacy-shell-invocation-functions) for details.