HowTo: Working with large files in Ruby efficiently.
Tags: ruby, programming, filesprocessing
Published: 2019-09-03 13:07:41

---
title: HowTo: Working with large files in Ruby efficiently.
published: true
description: How to process large files in Ruby while keeping CPU and memory usage in control?
tags: ruby, programming, files processing
cover_image: https://tjay.dev/content/images/size/w2000/2019/09/erwan-hesry-Q34YB7yjAxA-unsplash.jpg
canonical_url: https://tjay.dev/howto-working-efficiently-with-large-files-in-ruby/
---

## How can we read files in Ruby?

### * Testing file generated by running a following command:

```bash
❯ openssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -keyout key.pem -out cert.pem
```

It has a clearly defined the beginning and the end of the file which fill be useful while reading the files.


### 1. `File.read()` which is actually [`IO.read()`](https://ruby-doc.org/core-2.6.4/IO.html#method-c-read):

```ruby
> file = File.read("cert.pem")
=> "-----BEGIN CERTIFICATE-----\nMIICljCCAX4CCQD5x/0DnI1UazANBgkqhkiG9w0BAQsFADANMQswCQYDVQQGEwJQ\nTDAeFw0xOTA4MzExOTQ0NDdaFw0yOTA4MjgxOTQ0NDdaMA0xCzAJBgNVBAYTAlBM\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2qJrZayMFRE7zIeUL8CZ\nzqsOcwEv6flF41EjIvVf6h164i+NGkRu9E0wo1LHYsoF5tutYKKpRLJoY9xGq+Jr\n1SPOJYGBaFqKyQye+lnSzJdpnCAklXObfJpGtBmKCm4OTcb8eC4nm2q4x3mNkP5Z\nTgzdfIhALCwtD6wsHcyy5qmqGfPWAaGUDHqAQRu7QV/vu5VzJXgN0c6Zj+bOWw4H\n7Zu+FxtpUACQk4lnqt9CUzp6GX3dIETTfA3cpTFvoxwqBZGnrjsgZA5HzbyKRUYi\naigbkyzc701sJaS8gcjIKDy2s8L8MfqaJkMu+N52e5tXoj4oQT9wPzxOou+GpYM/\n4QIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQDDrOrN+asQjkOwjPcNLkycy4TJ/6QE\nraNDVZ1N5h+70vIQwmmCS+hBN7SSM0f0OxgEggvK0etNQb6LXWXAIa7pMuzhqmHR\n9Q/NBizj+GOIvH7EoCTVKYUkRLxEq5i63cm0ZvFu9qwr8v7IGM4HkLo3A0F6+Vcp\nGNuOBNcGqAtCXNhgcpzu/6zWT2kAj1M82IC4aCIiTGovDidnp2ZO4bV5PTCy7ecd\naeJxt9LIlt/FVk29sjdtutPMZgtQwKKp2gWyY9D7/x8Dxpf2DCkjAtqEdN3/GER6\nlybIrvAtYW7MNmu9MLkxionOak9CoZGsVg0kiXliHrhfxrDc8qLe8rqV\n-----END CERTIFICATE-----\n"
> file.bytesize
=> 956
> file.class
=> String
```
`read` method reads the entire file's content and assigns it to the variable as single String.


### 2. [`File.new()`](https://ruby-doc.org/core-2.6.4/File.html#method-c-new) and its synonym [`File.open()`](https://ruby-doc.org/core-2.6.4/File.html#method-c-open):

```ruby
> file = File.new("cert.pem")
=> #<File:cert.pem>
> lines = file.readlines
=> ["-----BEGIN CERTIFICATE-----\n",
 "MIICljCCAX4CCQD5x/0DnI1UazANBgkqhkiG9w0BAQsFADANMQswCQYDVQQGEwJQ\n",
 "TDAeFw0xOTA4MzExOTQ0NDdaFw0yOTA4MjgxOTQ0NDdaMA0xCzAJBgNVBAYTAlBM\n",
 "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2qJrZayMFRE7zIeUL8CZ\n",
 "zqsOcwEv6flF41EjIvVf6h164i+NGkRu9E0wo1LHYsoF5tutYKKpRLJoY9xGq+Jr\n",
 "1SPOJYGBaFqKyQye+lnSzJdpnCAklXObfJpGtBmKCm4OTcb8eC4nm2q4x3mNkP5Z\n",
 "TgzdfIhALCwtD6wsHcyy5qmqGfPWAaGUDHqAQRu7QV/vu5VzJXgN0c6Zj+bOWw4H\n",
 "7Zu+FxtpUACQk4lnqt9CUzp6GX3dIETTfA3cpTFvoxwqBZGnrjsgZA5HzbyKRUYi\n",
 "aigbkyzc701sJaS8gcjIKDy2s8L8MfqaJkMu+N52e5tXoj4oQT9wPzxOou+GpYM/\n",
 "4QIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQDDrOrN+asQjkOwjPcNLkycy4TJ/6QE\n",
 "raNDVZ1N5h+70vIQwmmCS+hBN7SSM0f0OxgEggvK0etNQb6LXWXAIa7pMuzhqmHR\n",
 "9Q/NBizj+GOIvH7EoCTVKYUkRLxEq5i63cm0ZvFu9qwr8v7IGM4HkLo3A0F6+Vcp\n",
 "GNuOBNcGqAtCXNhgcpzu/6zWT2kAj1M82IC4aCIiTGovDidnp2ZO4bV5PTCy7ecd\n",
 "aeJxt9LIlt/FVk29sjdtutPMZgtQwKKp2gWyY9D7/x8Dxpf2DCkjAtqEdN3/GER6\n",
 "lybIrvAtYW7MNmu9MLkxionOak9CoZGsVg0kiXliHrhfxrDc8qLe8rqV\n",
 "-----END CERTIFICATE-----\n"]
> lines.class
=> Array
```
`new` or `open` methods returns an instance of the `File` class on which we can call `readlines` method which reads the entire file's content, splits it line by line and returns an Array of Strings where one element is one line from the file.


### 3. `File.readlines()` which is actually [`IO.readlines()`](https://ruby-doc.org/core-2.6.4/IO.html#method-c-readlines):

```ruby
> lines = File.readlines("cert.pem")
=> ["-----BEGIN CERTIFICATE-----\n",
 "MIICljCCAX4CCQD5x/0DnI1UazANBgkqhkiG9w0BAQsFADANMQswCQYDVQQGEwJQ\n",
 "TDAeFw0xOTA4MzExOTQ0NDdaFw0yOTA4MjgxOTQ0NDdaMA0xCzAJBgNVBAYTAlBM\n",
 "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2qJrZayMFRE7zIeUL8CZ\n",
 "zqsOcwEv6flF41EjIvVf6h164i+NGkRu9E0wo1LHYsoF5tutYKKpRLJoY9xGq+Jr\n",
 "1SPOJYGBaFqKyQye+lnSzJdpnCAklXObfJpGtBmKCm4OTcb8eC4nm2q4x3mNkP5Z\n",
 "TgzdfIhALCwtD6wsHcyy5qmqGfPWAaGUDHqAQRu7QV/vu5VzJXgN0c6Zj+bOWw4H\n",
 "7Zu+FxtpUACQk4lnqt9CUzp6GX3dIETTfA3cpTFvoxwqBZGnrjsgZA5HzbyKRUYi\n",
 "aigbkyzc701sJaS8gcjIKDy2s8L8MfqaJkMu+N52e5tXoj4oQT9wPzxOou+GpYM/\n",
 "4QIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQDDrOrN+asQjkOwjPcNLkycy4TJ/6QE\n",
 "raNDVZ1N5h+70vIQwmmCS+hBN7SSM0f0OxgEggvK0etNQb6LXWXAIa7pMuzhqmHR\n",
 "9Q/NBizj+GOIvH7EoCTVKYUkRLxEq5i63cm0ZvFu9qwr8v7IGM4HkLo3A0F6+Vcp\n",
 "GNuOBNcGqAtCXNhgcpzu/6zWT2kAj1M82IC4aCIiTGovDidnp2ZO4bV5PTCy7ecd\n",
 "aeJxt9LIlt/FVk29sjdtutPMZgtQwKKp2gWyY9D7/x8Dxpf2DCkjAtqEdN3/GER6\n",
 "lybIrvAtYW7MNmu9MLkxionOak9CoZGsVg0kiXliHrhfxrDc8qLe8rqV\n",
 "-----END CERTIFICATE-----\n"]
> lines.class
=> Array
```
Here, we have the same output as in the previous example by calling just class method `readlines` on `File` class.


### 4. `File.foreach()` which is actually `IO.foreach()`:

```ruby
> file = File.foreach("./cert.pem")
=> #<Enumerator: ...>
> file.entries
=> ["-----BEGIN CERTIFICATE-----\n",
 "MIICljCCAX4CCQD5x/0DnI1UazANBgkqhkiG9w0BAQsFADANMQswCQYDVQQGEwJQ\n",
 "TDAeFw0xOTA4MzExOTQ0NDdaFw0yOTA4MjgxOTQ0NDdaMA0xCzAJBgNVBAYTAlBM\n",
 "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2qJrZayMFRE7zIeUL8CZ\n",
 "zqsOcwEv6flF41EjIvVf6h164i+NGkRu9E0wo1LHYsoF5tutYKKpRLJoY9xGq+Jr\n",
 "1SPOJYGBaFqKyQye+lnSzJdpnCAklXObfJpGtBmKCm4OTcb8eC4nm2q4x3mNkP5Z\n",
 "TgzdfIhALCwtD6wsHcyy5qmqGfPWAaGUDHqAQRu7QV/vu5VzJXgN0c6Zj+bOWw4H\n",
 "7Zu+FxtpUACQk4lnqt9CUzp6GX3dIETTfA3cpTFvoxwqBZGnrjsgZA5HzbyKRUYi\n",
 "aigbkyzc701sJaS8gcjIKDy2s8L8MfqaJkMu+N52e5tXoj4oQT9wPzxOou+GpYM/\n",
 "4QIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQDDrOrN+asQjkOwjPcNLkycy4TJ/6QE\n",
 "raNDVZ1N5h+70vIQwmmCS+hBN7SSM0f0OxgEggvK0etNQb6LXWXAIa7pMuzhqmHR\n",
 "9Q/NBizj+GOIvH7EoCTVKYUkRLxEq5i63cm0ZvFu9qwr8v7IGM4HkLo3A0F6+Vcp\n",
 "GNuOBNcGqAtCXNhgcpzu/6zWT2kAj1M82IC4aCIiTGovDidnp2ZO4bV5PTCy7ecd\n",
 "aeJxt9LIlt/FVk29sjdtutPMZgtQwKKp2gWyY9D7/x8Dxpf2DCkjAtqEdN3/GER6\n",
 "lybIrvAtYW7MNmu9MLkxionOak9CoZGsVg0kiXliHrhfxrDc8qLe8rqV\n",
 "-----END CERTIFICATE-----\n"]
> lines.class
=> Array
```
`foreach` method returns an Enumerator instance on which we call `entries` which returns an Array of String, again each element is a line from the file.

As we can see above there are many methods that allow us to read the file. However which one should we use and why? Let's create a large file and check those methods again!

## Which methods should we use to read large files?

### Generating our test file

At first, let's generate a large file with randomized data inside:

```ruby
require 'securerandom'
one_megabyte = 1024 * 1024

name = "large_1G"
size = 1000

File.open("./#{name}.txt", 'wb') do |file|
  size.times do
    file.write(SecureRandom.random_bytes(one_megabyte))
  end
end
```

- `w` -  Write-only, truncates existing file to zero length or creates a new file for writing.
- `b` - Binary file mode. Suppresses EOL <-> CRLF conversion on Windows. And sets external encoding to ASCII-BIT unless explicitly specified.

As the result we generated 1GB file:

```bash
❯ ls -lah
...
-rw-r--r--   1 user  user   1.0G Aug 31 22:10 large_1G.txt
```



### Defining our metrics and profilers

There are probably 2 the most important metrics that we would like to track in our experiment:

- **Time** - How long does it take to open and read the file?
- **Memory** - How much memory does it take to open and read the file?

Also there will be one additional metric describing how many objects were freed by Garbage Collector.

We can prepare simple profiling methods:

```ruby
# ./helpers.rb
require 'benchmark'

def profile_memory
  memory_usage_before = `ps -o rss= -p #{Process.pid}`.to_i
  yield
  memory_usage_after = `ps -o rss= -p #{Process.pid}`.to_i

  used_memory = ((memory_usage_after - memory_usage_before) / 1024.0).round(2)
  puts "Memory usage: #{used_memory} MB"
end

def profile_time
  time_elapsed = Benchmark.realtime do
    yield
  end

  puts "Time: #{time_elapsed.round(2)} seconds"
end

def profile_gc
  GC.start
  before = GC.stat(:total_freed_objects)
  yield
  GC.start
  after = GC.stat(:total_freed_objects)

  puts "Objects Freed: #{after - before}"
end

def profile
  profile_memory do 
    profile_time do 
      profile_gc do
        yield
      end
    end 
  end 
end
```



### Testing our methods for reading files

- `.read`

```ruby
file = nil
profile do
  file = File.read("large_1G.txt")
end

Objects Freed: 39
Time: 0.52 seconds
Memory usage: 1000.05 MB
```

- `.new` + `#readlines`

```ruby
file = nil
profile do
  file = File.new("large_1G.txt").readlines
end

Objects Freed: 39
Time: 4.19 seconds
Memory usage: 1298.4 MB
```

- `.readlines`

```ruby
file = nil
profile do
  file = File.readlines("large_1G.txt")
end

Objects Freed: 39
Time: 4.24 seconds
Memory usage: 1284.61 MB
```

- `.foreach`

```ruby
file = nil
profile do
  file = File.foreach("large_1G.txt").to_a
end

Objects Freed: 40
Time: 4.42 seconds
Memory usage: 1284.31 MB
```

The examples we can see above allowed us to read the whole file and store it in local memory as one String or as an Array of Strings (each line from the file as one element in the Array).

As we can see, it requires **at least as much memory as the size of the file**:

- one **String** - 1GB file requires 1GB of memory. 
- an **Array of Strings** - 1GB memory for file's content + additional memory for an Array (+- 300MB here). This approach has one advantage, we can access whichever line of the file we want as long as we know which line is it.

At this point we can see that the methods that we tested are not really efficient. The bigger the file, the more memory we need. In longer term this approach might lead to some serious consequences, even killing the application.

Now, we need to us ourselves a question. **Can we process our files line by line?** If so, then we can read our files in a different way:

- `.new` + `#each`

```ruby
file = nil

profile do
  file = File.new("large_1G.txt")
  file.each { |line| line }
end

Objects Freed: 4100808
Time: 2.08 seconds
Memory usage: 57.68 MB
```

- `.new` + `#advise` + `#each` 

```ruby
file = nil

profile do
  file = File.new("large_1G.txt")
  file.advise(:sequential)
  file.each { |line| line }
end

Objects Freed: 4100808
Time: 2.22 seconds
Memory usage: 55.71 MB
```
Calling [`#advise`](https://ruby-doc.org/core-2.6.4/IO.html#method-i-advise) method announces an intention to access data from the current file in a specific pattern. No major improvement here with using `#advise` method.

- `.new` + `#read` - reading chunk by chunk

```ruby
file = nil
chunk_size = 4096
buf = ""

profile do
  file = File.new("large_1G.txt")
  while buf = file.read(chunk_size)
    buf.tap { |buf| buf }
  end
end

Objects Freed: 256037
Time: 1.27 seconds
Memory usage: 131.64 MB
```
We defined the `chunk` as 4096 bytes and we read our file chunk by chunk. Depending on the structure of your file this approach might be useful.

- `.foreach` + `#each_entry`

```ruby
file = nil

profile do
  file = File.foreach("large_1G.txt")
  file.each_entry { |line| line }
end

Objects Freed: 4100809
Time: 2.22 seconds
Memory usage: 53.02 MB
```
Creating an Enumerator instance as `file` and reading file line by line using `each_entry` method.

First thing we can notice is that memory usage is way lower. Main reason for that is that we read the file line by line and when the line is processed then it's garbage collected. We can see that by the size of the *Objects Freed*, it's quite high.

We also tried to use here an `#advise` method which we can tell how we want to process our file. More about [`IO#advise`](https://ruby-doc.org/core-2.6.4/IO.html#method-i-advise) can be found in the documentation. Unfortunately, it didn't help us out here.

Except [`IO#each`](https://ruby-doc.org/core-2.6.4/IO.html#method-i-each) method we have also similar methods like [`IO#each_byte`](https://ruby-doc.org/core-2.6.4/IO.html#method-i-each_byte) (reading byte by byte),[`IO#each_char`](https://ruby-doc.org/core-2.6.4/IO.html#method-i-each_char) (reading char by char) and [`IO#each_codepoint`](https://ruby-doc.org/core-2.6.4/IO.html#method-i-each_codepoint).

In the example with reading by chunks ([`IO#read`](https://ruby-doc.org/core-2.6.4/IO.html#method-i-read)) the memory usage will vary depending on the chunk size. If you find this way useful you can experiment with the chunk size.

When using [`IO.foreach`](https://ruby-doc.org/core-2.6.4/IO.html#method-c-foreach) we operate on Enumerator which gives us a few more methods like: [`IO#each_entry`](https://ruby-doc.org/core-2.6.4/Enumerable.html#method-i-each_entry), [`IO#each_slice`](https://ruby-doc.org/core-2.6.4/Enumerable.html#method-i-each_slice), [`IO#each_cons`](https://ruby-doc.org/core-2.6.4/Enumerable.html#method-i-each_cons). There is also [`lazy`](https://ruby-doc.org/core-2.6.4/Enumerable.html#method-i-lazy) method which returns a [Enumerator::Lazy](https://ruby-doc.org/core-2.6.4/Enumerator/Lazy.html). Lazy Enumerator has a few additional methods which enumerate values only on an as-needed basis. If you don't need to read the entire file but, for example, looking for a particular line containing given expression then it might be worth to check it out.

I could finish the article at this point, but what if before we even start reading the file we need to decrypt it? Let's move further to the example.

### Decrypting large file and processing it line by line

#### Prerequisites

Before we decrypt the file we need to encrypt our generated large file. We are going to use AES with 256 bits key length with Cipher Block Chaining (CBC) as mode.

```ruby
cipher = OpenSSL::Cipher::AES256.new(:CBC)
cipher.encrypt
KEY = cipher.random_key
IV = cipher.random_iv
```

Now, let's encrypt out file:

```ruby
cipher = OpenSSL::Cipher::AES256.new(:CBC)
cipher.encrypt
cipher.key = KEY
cipher.iv = IV

file = nil
enc_file = nil

profile do
  file = File.read("large_1G.txt")
  enc_file = File.open("large_1G.txt.enc", "wb")
  enc_file << cipher.update(file)
  enc_file << cipher.final
end

file.close
enc_file.close

Objects Freed: 12
Time: 3.6 seconds
Memory usage: 1000.02 MB
```

Seems like encrypting is also a quite memory consuming task. Let's adjust the algorithm a little bit:

```ruby
cipher = OpenSSL::Cipher::AES256.new(:CBC)
cipher.encrypt
cipher.key = KEY
cipher.iv = IV

file = nil
enc_file = nil

profile do 
  buf = ""
  file = File.open("large_1G.txt", "rb")
  enc_file = File.open("large_1G.txt.enc", "wb")
  while buf = file.read(4096)
    enc_file << cipher.update(buf)
  end
  enc_file << cipher.final
end

file.close
enc_file.close

Objects Freed: 768048
Time: 5.05 seconds
Memory usage: 145.93 MB
```

By changing the algorithm to read and cipher the file by chunks made the task much less memory consuming.

#### Decrypt

All right, let's try to decrypt it now:

```ruby
decipher = OpenSSL::Cipher::AES256.new(:CBC)
decipher.decrypt
decipher.key = KEY
decipher.iv = IV

dec_file = nil
enc_file = nil

profile do 
  buf = ""
  enc_file = File.open("large_1G.txt.enc", "rb")
  dec_file = File.open("large_1G.txt.dec", "wb")
  while buf = enc_file.read(4096)
    dec_file << decipher.update(buf)
  end
  dec_file << decipher.final
end

dec_file.close
enc_file.close

Objects Freed: 768050
Time: 3.5 seconds
Memory usage: 152.12 MB
```

Now, let's compare our files whether we properly encrypted and decrypted it:

```bash
❯ diff large_1G.txt large_1G.txt.dec

```

No differences were found. We are good here!

We managed to lower the memory usage quite significantly. That's great! 

Treat this article as a toolset that you can use in your specific case.

**This article was originally posted on my personal dev blog: [https://tjay.dev/](https://tjay.dev/howto-working-efficiently-with-large-files-in-ruby/?utm_source=dev.to&utm_medium=footer-link&utm_campaign=reblogging)**

*Photo by [Erwan Hesry](https://unsplash.com/@erwanhesry) on [Unsplash](https://unsplash.com/search/photos/luggage)*