Working with big files in golang
Published: 2021-09-21 17:14:35

---
title: Working with big files in golang
published: true
date: 2021-03-13 23:00:00 UTC
tags: 
canonical_url: https://devmarkpro.com/working-big-files-golang
---

Today, I am going to show you how to read files in golang **line-by-line**. Let's imagine that have a `jsonl` file. What's [jsonl](https://jsonlines.org/)? it's **json lines** in a simple term, it's a file that each line of it represents a valid `json` object. So if we read the file line by line, we can Marshal/Unmarshal each line of it separately. Here's an example of a `jsonl` file.

Each line of this file represents the data of a world cup.

```
{"year":"2018","host":"Russia","winner":"France"}
{"year":"2014","host":"Brazil","winner":"Germany"}
{"year":"2010","host":"South Africa","winner":"Spain"}
{"year":"2006","host":"Germany","winner":"Italy"}

```

## Working with bufio.Scanner

So let’s read this file line by line most easily and conveniently. the easiest way to read a file (at least for me) is using the [scanner](https://golang.org/pkg/bufio/#Scanner) from the **bufio** package in the standard library. First, we need to create an instance with the `NewScanner` function which is a very familiar way for constructing the structs in golang. this function accepts a Reader interface as input, and the good news is `os.File` implemented this interface. It means, we can open a file and pass the pointer of the file to bufio.NewScanner. Let’s see it in action.

```
// first open the file
file, err := os.Open("/Users/Mark/fifa-winners.jsonl")
if err != nil {
    log.Fatalf("could not open the file: %v", err)
}
// don't forget to close the file.
defer file.Close()
// finally, we can have our scanner
scanner := bufio.NewScanner(file)

```

So, we have the scanner, we are ready to go... scanner has a function named **Scan**. this function moves the scanner to the next token. I'll tell you what it means but for now, let's say each time **Scan** called, we read one line of our file. So if we want to move the scanner all through the file, we'd call the scan function in an infinite loop! The question is **How do we know, we can break the loop?** It's easy! Scan returns true as the return unless it meets the end of the file.

```
for {
    if scanner.Scan() {
        // we have a new line in each iteration
        continue
    }
    // we are done let's break the loop
    break
}
// the rest of our spaghetti

```

This code works, but have you know we can say golang to keep a loop running until met a specific condition. All the code above can be as simple as the code below:

```
for scanner.Scan() {
    // we have a new line in each iteration
}
// the rest of our spaghetti

```

Well, Let's get back to the point! we can have our **bytes** or **string** in each line easily by calling `Bytes()` and `Text()` functions.

```
for scanner.Scan() {
  // b is an array of bytes ([]byte)
  b := scanner.Bytes()
  // s is string
  s := scanner.Text()
}

```

Frankly, These functions are the same! for example `string(scanner.Bytes())` will give you the same result and that's what exactly happens in the `Text()` function.

We read our file, so is the mission completed? **Not exactly** , because we didn't handle any error yet.

the scanner has another function called `Err()`. This function gives you the **first** error that happened during the scan process. It means, when the scanner trying to move through the file, the _Scan_ function returns false, if something bad happened. So our loop instantly breaks and we will out of the loop. Now we can get that error and deal with it.

If we want to know in each line of the file that error happened, we should use a traditional way, we know the scanner starts from the beginning of the file (line number 1) so we can define a variable outside of the loop that represents the line number and increase it in each iteration.

```
lineNumber := 0
for scanner.Scan() {
    lineNumber++
        fmt.Println(scanner.Text())
}
// the rest of our spaghetti
if err := scanner.Err(); err != nil {
    log.Fatalf("something bad happened in the line %v: %v", lineNumber, err)
}

```

another thing that we should consider about the `Err()` function is it ignores the `io.EOF` so if we will give an error, it's a REAL one!

Let's have a run:

```
➜ big-files (main) ✗ go run main.go
{"year":"2018","host":"Russia","winner":"France"}
{"year":"2014","host":"Brazil","winner":"Germany"}
{"year":"2010","host":"South Africa","winner":"Spain"}
{"year":"2006","host":"Germany","winner":"Italy"}

```

It worked, So what's next?

## Fix bufio.Scanner: token too long error

We said, each line of a jsonl file represents a valid json, so it could be too long. We also said the `Scan` function moves the scanner to the next token but the question is where is the next token!?

The scanner function has another method that is not as famous as its siblings, `Buffer` and it needs a buffer and an integer as input and you can set the maximum size of the buffer with this function.

**bufio** package has a maximum token size which equals `64 * 1024` (~65.6kb). So if one line of our lines is bigger than this size, we got this error `token too long error`.

We found the answer to our question: The next token is where the scanner reaches max size (default 65kb) **OR** the end of the line.

### Approach 1: Bigger buffer size

The first approach to tackling this problem is to increase the buffer size. Actually, the name `bufio.MaxScanTokenSize` is a little misleading because it's not the actual maximum it's **THE DEFAULT MAXIMUM** size. so we can increase it.

```
buf := []byte{}
scanner := bufio.NewScanner(file)
// increase the buffer size to 2Mb
scanner.Buffer(buf, 2048*1024)

```

Now we can process `jsonl` files with lines up to 2Mb. It's good but what if we need more? We can increase this number as much as we want (probably) but if our file has _5.000.000_ rows and just one of them is 100Mb we need to increase our scanner to this size just for one line, or use another approach!

### Approach2

the next way to read such a tough file! is using another function! Bufio (buffer-io) gave us way more than a simple scanner to work with files and we have to choose one of them based on our needs and requirements. in this case, Scanner cannot satisfy what we need so let's take a look at `ReadLine` function of `bufio.Reader`. It's a little bit lower level than the scanner. generally speaking, when you hear the word `lower-level` you should do more for simple things but you have more access and power!

So let's get started. First we need a reader:

```
reader := bufio.NewReader(file)

```

reader, has **ReadLine** function which _tries_ to read the entire line. Just like the scanner, we need to call this function in a for loop but since we are at the _lower level_! we don't have a nice-easy boolean in return anymore to know that we can break the loop.

the other difference is the error that we will give from the `ReadLine` function, which can also be `io.EOF`. It's not going to be a real error for us, so we have to handle it too.

```
reader := bufio.NewReader(file)
for {
    line, _, err := reader.ReadLine()
    if err != nil {
        if err == io.EOF {
            break
        }
        log.Fatalf("a real error happened here: %v\n", err)
    }
    fmt.Println(string(line))
}

```

As you probably already know, We just read the file so far, we did actually solve the problem that we had with the gigantic lines.

we ignore the second parameter that we gave from the ReadLine function and that one is what we exactly need to solve our problem. It's a boolean named `isPrefix`. If the line is too long and `ReadLine` cannot put all of its content in the buffer, It returns the filled buffer and set isPrefix to true which means we will give the **next part** of the line in the next call of the `ReadLine` function.

So we just need to call the ReadLine function until `isPrefix` becomes `false` then we can go for the next line of our file. You probably already noticed that we are talking about a recursive function. First I define the function that we want to call recursively.

```
func read(r *bufio.Reader) ([]byte, error) {
    var (
        isPrefix = true
        err error
        line, ln []byte
    )

    for isPrefix && err == nil {
        line, isPrefix, err = r.ReadLine()
        ln = append(ln, line...)
    }

    return ln, err
}

```

`isPrefix` is true at the first place and error is also nil so we make sure the for loop will run at least one time. It behaves like the do-while loop. We re-assign variables inside the loop so we call `r.ReadLine` unless we got an error OR `isPrefix` is false. in each iteration, we append the bytes that we get from `r.ReadLine()` to another variable. Now it's time to call this function inside the main function.

```
reader := bufio.NewReader(file)
for {
    line, err := read(reader)
    if err != nil {
        if err == io.EOF {
            break
        }
        log.Fatalf("a real error happened here: %v\n", err)
    }
    fmt.Println(string(line))
}

```

That's it! We solve the problem. here's the complete code:

```
package main

import (
    "bufio"
    "fmt"
    "io"
    "log"
    "os"
)

func main() {
    // first open the file
    file, err := os.Open("./fifa-winners.jsonl")
    if err != nil {
        log.Fatalf("could not open the file: %v", err)
    }
    defer file.Close()
    log.Println(" *******************READ WITH SCANNER*******************")
    readWithScanner(file)
    log.Println("*******************READ WITH READLINE()*******************")

    // we just reset the offset. because we read this file once
    // imagine the cursor is in the end of the file so we have to get back to the first line and read it again 
    file.Seek(0, 0)
    readWithReadLine(file)

    log.Println("we read a file twice!")
}

// Read with simple scanner

func readWithScanner(file *os.File) {
    // first open the file
    file, err := os.Open("./fifa-winners.jsonl")
    if err != nil {
        log.Fatalf("could not open the file: %v", err)
    }
    // finally, we can have our scanner
    buf := []byte{}
    scanner := bufio.NewScanner(file)
    scanner.Buffer(buf, 2048*1024)
    lineNumber := 1
    for scanner.Scan() {
        fmt.Println(scanner.Text())
        lineNumber++
    }
    // the rest of our spaghetti
    if err := scanner.Err(); err != nil {
        log.Fatalf("something bad happened in the line %v: %v", lineNumber, err)
    }
}

// Read with Readline function

func read(r *bufio.Reader) ([]byte, error) {
    var (
        isPrefix = true
        err error
        line, ln []byte
    )

    for isPrefix && err == nil {
        line, isPrefix, err = r.ReadLine()
        ln = append(ln, line...)
    }

    return ln, err
}

func readWithReadLine(file *os.File) {
    reader := bufio.NewReader(file)
    for {
        line, err := read(reader)
        if err != nil {
            if err == io.EOF {
                break
            }
            log.Fatalf("a real error happened here: %v\n", err)
        }
        fmt.Println(string(line))
    }
}

```