5 log parsing commands
Tags: linux, beginners, development, programming
Published: 2024-06-23 06:00:05

Have you ever tried to find something in the server log file? While downloading and opening the file in an editor might seem straightforward, it's often time-consuming and unproductive. Instead, using command-line tools can be more efficient and effective. Here are some common commands you should try.

The practice file today is system.log

```prolog
2024-06-12 13:39:30 [INFO] Server started on port 8080
2024-06-12 13:40:12 [ERROR] Failed to connect to database
2024-06-12 13:41:05 [INFO] User 'john_doe' logged in
2024-06-12 13:42:16 [WARNING] Disk space low on /dev/sda1
2024-06-12 13:43:27 [INFO] Scheduled job 'backup' started
2024-06-12 13:44:38 [ERROR] Could not complete backup: disk full
2024-06-12 13:45:49 [INFO] User 'jane_smith' logged out
2024-06-12 13:46:50 [INFO] Server shutdown initiated
2024-06-12 13:47:51 [INFO] Server stopped
2024-06-12 13:48:52 [INFO] Server started on port 8080
2024-06-12 13:49:53 [INFO] User 'john_doe' logged in
2024-06-12 13:50:54 [ERROR] Failed to retrieve data from API
2024-06-12 13:51:55 [WARNING] High memory usage detected
2024-06-12 13:52:56 [INFO] Scheduled job 'cleanup' started
2024-06-12 13:53:57 [ERROR] Cleanup job failed: permission denied
2024-06-12 13:54:58 [INFO] User 'john_doe' logged out
2024-06-12 13:55:59 [INFO] Server shutdown initiated
2024-06-12 13:56:00 [INFO] Server stopped
2024-06-12 13:57:01 [INFO] Server started on port 8080
2024-06-12 13:58:02 [ERROR] Failed to connect to database
2024-06-12 13:59:03 [INFO] User 'jane_smith' logged in
2024-06-12 14:00:04 [WARNING] Disk space low on /dev/sda1
2024-06-12 14:01:05 [INFO] Scheduled job 'backup' started
2024-06-12 14:02:06 [ERROR] Could not complete backup: disk full
2024-06-12 14:03:07 [INFO] User 'jane_smith' logged out
2024-06-12 14:04:08 [INFO] Server shutdown initiated
2024-06-12 14:05:09 [INFO] Server stopped
2024-06-12 14:06:10 [INFO] Server started on port 8080
2024-06-12 14:07:11 [INFO] User 'john_doe' logged in
2024-06-12 14:08:12 [ERROR] Failed to retrieve data from API
2024-06-12 14:09:13 [WARNING] High memory usage detected
2024-06-12 14:10:14 [INFO] Scheduled job 'cleanup' started
2024-06-12 14:11:15 [ERROR] Cleanup job failed: permission denied
2024-06-12 14:12:16 [INFO] User 'john_doe' logged out
2024-06-12 14:13:17 [INFO] Server shutdown initiated
2024-06-12 14:14:18 [INFO] Server stopped
```

## 1. Display the Contents of the Log File

### `cat` Command

- **Purpose**: used to display the content of files.
- **Usage**: `cat filename`
- **Example**: `cat server.log`

```prolog
$ cat system.log
2024-06-12 13:39:30 [INFO] Server started on port 8080
2024-06-12 13:40:12 [ERROR] Failed to connect to database
2024-06-12 13:41:05 [INFO] User 'john_doe' logged in
2024-06-12 13:42:16 [WARNING] Disk space low on /dev/sda1
2024-06-12 13:43:27 [INFO] Scheduled job 'backup' started
2024-06-12 13:44:38 [ERROR] Could not complete backup: disk full
2024-06-12 13:45:49 [INFO] User 'jane_smith' logged out
2024-06-12 13:46:50 [INFO] Server shutdown initiated
2024-06-12 13:47:51 [INFO] Server stopped
2024-06-12 13:48:52 [INFO] Server started on port 8080
2024-06-12 13:49:53 [INFO] User 'john_doe' logged in
2024-06-12 13:50:54 [ERROR] Failed to retrieve data from API
2024-06-12 13:51:55 [WARNING] High memory usage detected
2024-06-12 13:52:56 [INFO] Scheduled job 'cleanup' started
2024-06-12 13:53:57 [ERROR] Cleanup job failed: permission denied
2024-06-12 13:54:58 [INFO] User 'john_doe' logged out
2024-06-12 13:55:59 [INFO] Server shutdown initiated
2024-06-12 13:56:00 [INFO] Server stopped
2024-06-12 13:57:01 [INFO] Server started on port 8080
2024-06-12 13:58:02 [ERROR] Failed to connect to database
2024-06-12 13:59:03 [INFO] User 'jane_smith' logged in
2024-06-12 14:00:04 [WARNING] Disk space low on /dev/sda1
2024-06-12 14:01:05 [INFO] Scheduled job 'backup' started
2024-06-12 14:02:06 [ERROR] Could not complete backup: disk full
2024-06-12 14:03:07 [INFO] User 'jane_smith' logged out
2024-06-12 14:04:08 [INFO] Server shutdown initiated
2024-06-12 14:05:09 [INFO] Server stopped
2024-06-12 14:06:10 [INFO] Server started on port 8080
2024-06-12 14:07:11 [INFO] User 'john_doe' logged in
2024-06-12 14:08:12 [ERROR] Failed to retrieve data from API
2024-06-12 14:09:13 [WARNING] High memory usage detected
2024-06-12 14:10:14 [INFO] Scheduled job 'cleanup' started
2024-06-12 14:11:15 [ERROR] Cleanup job failed: permission denied
2024-06-12 14:12:16 [INFO] User 'john_doe' logged out
2024-06-12 14:13:17 [INFO] Server shutdown initiated
2024-06-12 14:14:18 [INFO] Server stopped
```

## 2. Search for lines

### `grep`

- **Purpose**: `powerful command` for searching text using patterns, and filtering log entries based on specific criteria.
- **Usage**: `cat filename | grep “filter-condition”` or `grep condition` filename
- **Example**: `grep "ERROR" server.log`

```prolog
$ grep "ERROR" system.log

2024-06-12 13:40:12 [ERROR] Failed to connect to database
2024-06-12 13:44:38 [ERROR] Could not complete backup: disk full
2024-06-12 13:50:54 [ERROR] Failed to retrieve data from API
2024-06-12 13:53:57 [ERROR] Cleanup job failed: permission denied
2024-06-12 13:58:02 [ERROR] Failed to connect to database
2024-06-12 14:02:06 [ERROR] Could not complete backup: disk full
2024-06-12 14:08:12 [ERROR] Failed to retrieve data from API
2024-06-12 14:11:15 [ERROR] Cleanup job failed: permission denied
```

## 3. Display Lines with Customize Condition

### `awk`

- **Purpose**: Introduce `awk` as a powerful text processing tool, ideal for manipulating data and generating reports.
- **Usage**: `awk condition filename`
- **Example 1**: Display lines with timestamps between `13:50:00` and `14:00:00`:

```prolog
$ awk '/13:5[0-9]:[0-9][0-9]/ || /14:00:00/' system.log

2024-06-12 13:50:54 [ERROR] Failed to retrieve data from API
2024-06-12 13:51:55 [WARNING] High memory usage detected
2024-06-12 13:52:56 [INFO] Scheduled job 'cleanup' started
2024-06-12 13:53:57 [ERROR] Cleanup job failed: permission denied
2024-06-12 13:54:58 [INFO] User 'john_doe' logged out
2024-06-12 13:55:59 [INFO] Server shutdown initiated
2024-06-12 13:56:00 [INFO] Server stopped
2024-06-12 13:57:01 [INFO] Server started on port 8080
2024-06-12 13:58:02 [ERROR] Failed to connect to database
2024-06-12 13:59:03 [INFO] User 'jane_smith' logged in
```

- **Example 2**: Extract and print the date and time of each entry

```prolog
$ awk '{print $1, $2}' system.log

2024-06-12 13:39:30
2024-06-12 13:40:12
2024-06-12 13:41:05
2024-06-12 13:42:16
2024-06-12 13:43:27
2024-06-12 13:44:38
2024-06-12 13:45:49
2024-06-12 13:46:50
2024-06-12 13:47:51
2024-06-12 13:48:52
2024-06-12 13:49:53
2024-06-12 13:50:54
2024-06-12 13:51:55
2024-06-12 13:52:56
2024-06-12 13:53:57
2024-06-12 13:54:58
2024-06-12 13:55:59
2024-06-12 13:56:00
2024-06-12 13:57:01
2024-06-12 13:58:02
2024-06-12 13:59:03
2024-06-12 14:00:04
2024-06-12 14:01:05
2024-06-12 14:02:06
2024-06-12 14:03:07
2024-06-12 14:04:08
2024-06-12 14:05:09
2024-06-12 14:06:10
2024-06-12 14:07:11
2024-06-12 14:08:12
2024-06-12 14:09:13
2024-06-12 14:10:14
2024-06-12 14:11:15
2024-06-12 14:12:16
2024-06-12 14:13:17
2024-06-12 14:14:18
```

## 4. Sort Log Entries

### `sort`

- **Purpose**: Sort lines in text files.
- **Usage**: Sort log entries by date, time, or any other field.
- **Example**: `cat system.log | awk '{print $1, $2, $3}' | sort`

```prolog
$ cat system.log | awk '{print $1, $2, $3}' | sort
2024-06-12 13:39:30 [INFO]
2024-06-12 13:40:12 [ERROR]
2024-06-12 13:41:05 [INFO]
2024-06-12 13:42:16 [WARNING]
2024-06-12 13:43:27 [INFO]
2024-06-12 13:44:38 [ERROR]
2024-06-12 13:45:49 [INFO]
2024-06-12 13:46:50 [INFO]
2024-06-12 13:47:51 [INFO]
2024-06-12 13:48:52 [INFO]
2024-06-12 13:49:53 [INFO]
2024-06-12 13:50:54 [ERROR]
2024-06-12 13:51:55 [WARNING]
2024-06-12 13:52:56 [INFO]
2024-06-12 13:53:57 [ERROR]
2024-06-12 13:54:58 [INFO]
2024-06-12 13:55:59 [INFO]
2024-06-12 13:56:00 [INFO]
2024-06-12 13:57:01 [INFO]
2024-06-12 13:58:02 [ERROR]
2024-06-12 13:59:03 [INFO]
2024-06-12 14:00:04 [WARNING]
2024-06-12 14:01:05 [INFO]
2024-06-12 14:02:06 [ERROR]
2024-06-12 14:03:07 [INFO]
2024-06-12 14:04:08 [INFO]
2024-06-12 14:05:09 [INFO]
2024-06-12 14:06:10 [INFO]
2024-06-12 14:07:11 [INFO]
2024-06-12 14:08:12 [ERROR]
2024-06-12 14:09:13 [WARNING]
2024-06-12 14:10:14 [INFO]
2024-06-12 14:11:15 [ERROR]
2024-06-12 14:12:16 [INFO]
2024-06-12 14:13:17 [INFO]
2024-06-12 14:14:18 [INFO]
```

## 5. Unique the display result

### `uniq`

- **Purpose**: Describe how `uniq` removes or counts duplicate lines.
- **Usage**: `cat filename | uniq -c`
- **Example**: `cat server.log | grep "ERROR" | awk '{print $4}' | sort | uniq -c`

```prolog
cat system.log | grep "ERROR" | awk '{print $4}' | sort | uniq -c
   2 Cleanup
   2 Could
   4 Failed
```

## Final thought

Analyzing information files is crucial for system administration, troubleshooting, and monitoring. Using a combination of command-line tools like `cat`, `grep`, `awk`, `sort`, and `uniq`, you can effectively manage and extract valuable insights from your log files.