CLI - A comprehensive text-based tutorial
Published: 2022-04-28 02:11:26

Whether you're experienced in the Information Technology field or if you're still a begginer, surely you've heard about consoles or terminals, or even used one. But what really are they and what are they really capable of? Aren't terminals just those dark and minimalist screens in which only the "tech wizards" can find meaning? Can they be useful in something modern like Data Science?

We'll try to conceptualize what is the CLI and what is the importance of knowing its mechanisms in this current tutorial, with applications to text processing, showing how some of its functionalities can be useful to a Data Science professional, and how powerful this simple tool can be.

# Conceptualizing the CLI

OK, just above we mentioned that mostly everyone knows about terminals. Let's then conceptualize it. In the not-so-distant past, computers were really expensive and not a thing that you'd have in your house, as they were only owned by big enterprises. 

Multiple employees shared the computer in their activities, and they operated the computer through the Command-Line Interfaces (CLI). The CLI was a text-only interface in which users would use the computer's resources typing whatever available functionalities, called commands, which were interpreted by a program called Command Language Interpreter, and then ran into the computer. These instructions were typed via the terminal, a hardware device used to enter and "translate" data into a computer.

OK, so now we know what the CLI and the terminal are, but they seem a bit antique, so one might ask: what's the use of this? Well, nowadays we of course have computers on our homes, and you can make use of them without the need to type every action, because we now have the Graphical User Interfaces (GUI). And the evolution of computers hasn't given up on terminals yet, as they are now emulated within GUIs with the terminal emulators, so we can still access all its powerful tools within a modern and advanced context in which our computer technology is stepping on now.

In the next section, we'll get into a hands-on approach, learning about some useful and powerful commands, most of them applied to text-processing. These tools will prove useful even in modern context like Data Science (we'll se that most of them are quite useful in data processing). 

For all the examples here, we'll be using a Unix shell, as Unix systems are the most common in Data Science fields. All the examples were run in a Z-Shell (ZSH) and are Bash-portable (another type of shell).

# Hands-on

## Basic Filesystem

We first need to know how to navigate in the shell to get our folders, files and such. 

The first commands we'll learn are about working directory (the directory our terminal prompt is currently working on): the command `pwd` print the name of the current working directory, and the command `cd`, acronym for "change directory", will, just as the name says, change our working directory based on the path we pass to it. For example, `cd folder/child_folder` will change our working directory to `child_folder`, because that's what the path pointed at.

To see the contents of the current folder, we can type `ls`, and it'll show the files, child folders and such. You can also pass a path as parameter to this command, so it'll print the contents inside it. There are several options we can combine within `ls` to get more information about the files, and you can search about them if you ever need.

## Glob Patterns

Similarly as we can do in regex patterns, we can match patterns in folder, looking for filenames and such that match our patterns, called glob patterns. They're a powerful tool we can use to modify the filesystem, but it's as dangerous as it's powerful, so we need to be very careful when executing operations. 

We'll not spend much time talking about these patterns; instead we'll offer a list of wildcards used to construct these patterns. They'll work just like a regex, so try it on!
 
- `?` match any single character
- `*` match any string (many characters)
- `[list_of_char]` match any char contained in the list, example: `[aeiou]` will match vowels
- `[!list_of_char]` match any not contained in the list
- `[[:alpha:]]` any letter
- `[[:digit:]]` any number
- `[[:alnum:]]` alphanumeric
- `[[:lower:]]` lowercase letters
- `[[:upper:]]` uppercase letters

## File inspection and printing

Well, suppose you have a file, a dataset. This dataset is saved in a text file format, say, a CSV. You want to get a feeling on what this file is containing, but the file is too big and you only want to inspect it briefly. Well, you can simply use the commands `head` and `tail`. Let's inspect them.

The `head filename` command will, by default, print the first 10 lines in the file. We can, of course, change this, by adding the option in the command, outputting as many lines as we want: `head -n 15 filename` will print the first 15 lines in the file.

The `tail filename` is the same thing, with the same options, but it'll print the last lines, not the first ones.

Now, say we want to pretty print this file. We want to display it on the terminal the same way we'd see in a table. For this, we can use the commando `column`. As stated above, our file is in CSV format, so its separators are commas. The `column` command needs us to specify the separator for the file, otherwise it wont work. So, for pretty printing it, we call `column -s"," -t filename`, the `-s` option being for specifying the separator and the `-t` to print it in tabular style. Try it out!

Now, we see that our dataset is far too big to work on. We need a random sample of reduced size to work on it. We can call the command `shuf` to sample our file. Calling `shuf -n 150 filename` will extract 150 random rows from our file. Pretty useful for inspecting the contents in a more generalized way than `head` or `tails`.

Finally, to know how many words, lines and characters (in this order) a file has, we can use `wc filename`.

We can see that these commands resemble a lot the Pandas commands, so they're pretty useful for Data Science! For those who don't know, Pandas is the most used Python library in the context of Data Science, providing methods for working with datasets in the same way we did in this section and some more things. Check it out if you're curious!

## Text processing

In this section, we'll cover a critical skill to Data Science within the shell context: text processing. It is whatever we can do with text, be it reformatting, modifying or extracting patterns.

The first command is the `cat` command. It'll concatenate as many files as we want. All we have to do is type `cat file1 file2 file3...` and the shell will print on the screen the concatenated files, in the passed order. If we want to just print the contents of the file on the screen, we can just use `cat filename`, and as we're not concatenating it with anything, it'll print itself.

The next command is `sort`. As the name indicates, it'll sort the file. Don't worry though, for it'll not change your file, just print on the screen the sorted result. Let's see a list of examples:

- `sort filename` is the default sorting
- `sort -r filename` sorts in reverse order
- `sort -u filename` sorts and removes duplicates
- `sort -t"," -k1,1 -k3,3 -k5-7` will sort using commas as separators, by columns 1, 3 and 5 through 7
- `sort -t"," -k2,2gr` will sort by column 2, and numerically (option `g`).

To select columns of a file, we use the command `cut`. Let's learn by example:

- `cut -d"," -f2-9` will select lines from 2 to 9, using the comma as separator

To select lines matching a glob pattern, we can use `grep`. Simply type `grep pattern filename` for it to catch the lines matching the patterns in the file. We can use some options to fine-tune it accordingly: `-n` will number the lines, `-i` will ignore case and `-v` will revert the match.

## Redirection and Pipelines

OK, so now we have some useful commands for text processing. So, in this section, we'll cover how to redirect the output of these commands in a way we can either combine them with another command or generate a file output. Let's see it.

First of all, the output of a command is printed on the terminal screen because it is default to do so. We can redirect the output in a way its not printed on the screen (which is sometimes not that useful), but on a new file! Say, for example, that we have a file contaning a CSV dataset, but we have far too many rows on it. We can, as learned previously, sample it. But this will print on the screen, and that's not helpful at all if we want to work on the sampled dataset. 

We can solve this by redirecting the output to a new file, so we can work on this file later. We do so by typing: `shuf -n 200 bigger_file >sampled_file`. With the use of the operator `>`, we sampled the dataset and saved it on a new file. Don't worry if the file doesn't exist, the redirect will create it for you. Do be mindful, though, that this will overwrite the contents of the file if it exists. If you want to append to a file without overwriting it, just use `>>` instead.

Now, let's say we want to combine the use of sampling and pretty printing. We could, as we just learned, save the sampled part in a file and pretty print it. But this would leave us with an extra file, a waste of space, especially if we just wanted to see the sample and not work with it. For combining two commands by passing the output of the first as input for the second, we use the pipe operator. By typing `shuf -n 15 filename | column -s"," -t filename`, we use the output of the `shuf` to work as an input for `column`, which'll print the dataset (assuming it's comma-separated) in tabular form. We got what we wanted without the need of extra space. Pretty neat, don't you think?

With the power of our many options within the shell commands, we can now combine then and redirect its outputs, making the use of them even more powerful than before.

# Conclusion

So, by learning all of that, we can easily see that the CLI is not some antique tool used only in the past, but a powerful tool that has functionalities that can fit in a modern context.

Within the context of text processing, we learned a variety of things that can be used in the shell that have some use in Data Science tasks. It's nice to use those tools because they're quite fast and demand a lot less work than producing a Python script to do the same, for example. 

Don't stop here and keep learning, the CLI has a lot to offer even if you're working on a modern context like Data Science!