Linux Ninja: Master Text Manipulation from Beginner to Badass in the CLI
Tags: cli, beginners, basic, bash
Published: 2024-05-03 21:28:29

## Standard out

From your command line you are able to create a new file and insert text into it.

```bash
$ echo Hello from the command line > test.txt
```

If you try this command in your terminal, you'll notice a new file named `test.txt` was created and if you open this file, you'll see the text `Hello from the command line` inside of it.

`echo` prints the text `Hello from the command line` to the screen.
By default the `echo` takes the input (standard input or stdin) from the keyboard and returns the output (standard output or stdout) to the screen.

`>` is a redirection operator that allows us to send the standard output to go into a file.

if you want to add more text to the file `test.txt` you can use the `>>` to append text to it instead of overwriting the file.

## Standard input

Just like the standard output redirection operator `>` we can also use the is operator as well `<` and what is it called? yes you guessed it! Standard input redirection operator!

Let's look at an example

```bash
$ cat < test.txt > readme.txt
```

if you don't know what the `cat` command does Check out my article [17 commands to master the Linux Command Line](https://dev.to/fullstackdev/master-the-linux-command-line-1aj)

Here we redirected `test.txt` file to be our standard input and the output of this file `Hello from the command line` will be redirecte to the `readme.txt`

so if we tried to read the `readme.txt` file it will have `Hello from the command line` as output

```bash
$ cat readme.txt
```

## Standard Error

So far we've seen 2 different streams standard input (stdin) that receives data as input `<` and standard output which is what you see on the terminal and can be redirected to a file using `>`.

Standard Error or stderr is specifically for error messages produced by program on the terminal even if stdout is redirected. Here's an example:

```bash
$ ls /nonexistent_directory/ > test.txt
```

The error will be `No such file or directory`

You can redirect an error message to the file by using file descriptors.
What are file descriptors you may ask? Well a file descriptor is a positive number that is used to access a file. The file descriptors for stdin stdout and stderr is 0 1 and 2 respectively.

To redirect the error message to a file:

```bash
$ ls /nonexistent_directory/ 2> test.txt
```

The error message should be inside `test.txt`

We can also send the error message to the file and the result of `ls` with it as well. (even if the directory doesn't exist there might be a minimal output which is not common). Common output will be an empty string or a new line.

```bash
$ ls /nonexistent_directory/ > test.txt 2 > &1
```

if we want to get rid of sderr messages you can use this command

```bash
$ ls /nonexistent_directory/ 2 > /dev/null
```

## Pipe and Tee

The pipe operator is represented by `|`, it acts as a connector by taking the output of a command and sending it as the input to another program.

let's list the details about the `dev` directory.

```bash
$ ls -la /dev/
```

Now you can use the pipe and the 'less' command to navigate through the list using the up and down arrow keys.

```bash
$ ls -la /dev/ | less
```

You can send the output of of the command to a file. You can write of this command to two streams which will allow you to see the result of the command on the terminal and then in the file.

```bash
$ ls -la /dev/ | tee my_output.txt
```

## Environment

An environment variable (PATH) exists on your system. When you write a command name this variable informs the computer where to look for the programs.
If you download a program and place it in a directory where the variable PATH does not check and you try to use the tool it by typing its name in the terminal, what typically happens is thet the computer searches for it in the listed directory in PATH and since where you placed it is not checked by path you will get `command not found`

```bash
$ echo $HOME
$ echo $USER
$ echo $PATH
```

The home directory path, your username and a list of path separated by colons that your system searches when it runs a command will be displayed on the terminal respectively.

# cut

The command `cut` is used for extracting pieces of text from files. To grab the pieces of data the `cut` command does it based on `fields` which is a descrete pieve of data within a line such as a comma, tab, semicolon etc...
We use the flag `-f` which stands for fields to tell `cut` what to extract. By default `cut` uses the tab as delimiters to seperate fields.

For example, we created a file named `text.txt` with some data in it

```bash
$ echo 'The bear jumped over the fence; The   snake followed it to the forest' > text.txt

$ cut -f 2 text.txt
```

The command using cut will extract the word `snake`. Let's break it down further.
We told `cut` to use the default delimiter which is tab by using -f and not specifying a delimiter and `2` stands for the second column which is what comes after the `;` semicolon. since the word snake is preceeded with tab then it will be extracted.

Now let's specify a delimiter other than tab.

```bash
$ cut -f 1 -d ";" text.txt
```

Here we're telling cut to select the first column `1` and assigned a new delimiter instead of tab which is the `;` by using the `-d` flag. The output will be `The bear jumped over the fence`.

## Paste

The paste command appends lines in a file for example we have a file named `testing.txt`

```
Hello

 I am

 Learning

 How to use

 Linux commands
```

Let's say we want all the content of `testing.txt` merged into one line

```bash
$ paste -d ' ' -s testing.txt
```

Here we specified the the delimiter to be a space. There is a space from the second line down to the end in `testing.txt`. `paste` will make the `testing.txt` a oneliner by bringing each sentence preceeded by a space to be appended to the first sentence.

## Head

The `head` command will show you the first 10 lines in a file by default. You can also modify how many lines you want to see. For example you can choose to see 20 lines instead of 10 by using the `-n` flag (number of lines).

```bash
$ head path_of_file
$ head -n 20 path_of_file
```

## Tail

The `tail` command is the same as the `head` command but it shows you the last 10 lines of a file instead of the first 10.

## Expand and unexpand

Let's say you want to convert tabs in a file to spaces

```bash
$ expand name_of_file.txt
```

If you want to convert the spaces back to tabs you can use this command

```bash
$ unexpand -a name_of_file.txt
```

## Joining and splitting files

If you have two text files you want to combine you can use this command

```bash
$ join text_file_1.txxt text_file_2.txt
```

They will be joined by fields meaning the first line of both files will be on the first line, the second line of both files will be joined on the second line etc..

This command will split a file into different files the default naming of the files will start with `x` followed by alphabetical suffixes `aa, ab` etc...

- `-a #`: Sets the number of characters used in the suffix (default is 2).

```bash
$ join -t ',' accounts.txt logged.txt
```

- `-t ','`: defines the delimiter which is separating the fields

The purpose of `join` is to merge lines from multiple files based on a matching filed. For example both of your files could have numbered lines and will be merged based on that.

You can split by size (bytes `-b`) or by lines `-l`

```bash
$ split -b 1000 name_of_file.txt
$ split -l 20 name_of_file.txt
```

## Sort

The `sort` command sorts lines in a file by alphabelitcal order

```bash
$ sort file_name.txt
```

You can also sort in reverse alphabetical order

```bash
$ sort -r filename.txt
```

We can also numerically sort by using the `-n` flag. If you've got a file with words in it, it will be sorted based on the `ASCII` value of the first character in each word.

Here's an example `fruits.txt`

```
Mango
Kiwi
Date
Pear
Banana
```

The `ASCII` value for the first character of each fruit is 77, 75, 68, 80, 66 respectively.

Now when we use it to numerically sort

```bash
$ sort -n fruits.txt
```

The output

```
Banana
Date
Kiwi
Mango
Pear
```

This command is also used to sort numbers inside a file. Example:

```
25
9
peach
5
```

The output

```
5
9
25
peach
```

## Translate

The `tr` command is used to translate a set of characters into another.
For example let's convert from lower to uppercase.

```bas
$ tr a-z A-Z

hello

HELLO
```

## Unique

The `uniq` command allows you to remove duplicates within a file

```bash
$ uniq file.txt
```

Using the `-c` flag you can get how many occurances each word has in the file

```bash
$ uniq -c file.txt
```

You can also find the unique value inside a file.

```bash
$ uniq -u file.txt
```

You can also know what words are the duplicates inside a file.

```bash
$ uniq -d text.txt
```

## Word count

You can know the total word count in a file by using the `wc` coommand

```bash
$ wc file.txt
```

This displays the lines, number of words and number of bytes respectively. You can select which one you want to see by using these flags

- `-l`: number of lines
- `-w`: number of words
- `-c`: number of bytes

`nl` is another command you can use to check the number of lines in a file

```bash
$ nl file.txt
```

## grep

It allows you to search files for characters that match a certain pattern.

You want to know if a word exists in a file?

```bash
$ grep hello testing.txt
```

This will find the word hello in the file `testing.txt`

you can use regular expression in your pattern so you can add the `-i` flag directly before the pattern you're looking for if it is case sensitive.

```bash
$ grep -i [a-zA-Z0-9]{6,10} file.txt
```

In this example we're trying to find a string that has a length between 6 and 10 characters and is alphanumeric and is case sensitive.