Under the Hood: “Slurping” and Streaming Files in Ruby
Tags: ruby, learning, beginners, programming
Published: 2018-07-10 13:46:45

---
title: "Under the Hood: “Slurping” and Streaming Files in Ruby"
layout: layout_ruby_magic
description: Streaming files line by line is often preferred over “slurping” the whole content at once. In this edition of Ruby Magic, we'll learn how streaming files and other I/O streams works in Ruby.
cover_image: https://blog.appsignal.com/images/blog/2018-07/slurping-streaming.jpg
tags: ruby, learning, beginners, programming
published: true
canonical_url: https://blog.appsignal.com/2018/07/10/ruby-magic-slurping-and-streaming-files.html
---

In this edition of Ruby Magic, we'll learn about streaming files in Ruby, how the `IO` class handles reading files without completely loading them into memory, and how it reads files per line by buffering read bytes. Let's dive right in!

## “Slurping” and Streaming Files

Ruby's `File.read` method reads a file and returns its full content.

``` irb
irb(main):001:0> content = File.read("log/production.log")
=> "I, [2018-06-27T16:45:02.843719 #9098]  INFO -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22] Started GET \"/articles\" for 127.0.0.1 at 2018-06-27 16:45:02 +0200\nI, [2018-06-27T16:45:02.846719 #9098]  INFO -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22] Processing by ArticlesController#index as HTML\nI, [2018-06-27T16:45:02.848212 #9098]  INFO -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22]   Rendering articles/index.html.erb within layouts/application\nD, [2018-06-27T16:45:02.850020 #9098] DEBUG -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22]   Article Load (0.3ms)  SELECT \"articles\".* FROM \"articles\"\nI, [2018-06-27T16:45:02.850901 #9098]  INFO -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22]   Rendered articles/index.html.erb within layouts/application (1.7ms)\nI, [2018-06-27T16:45:02.851633 #9098]  INFO -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22] Completed 200 OK in 5ms (Views: 3.4ms | ActiveRecord: 0.3ms)\n"
```

Internally, this opens the file, reads its content, closes the file, and returns the content as a single string. By "slurping" the file's content at once, it's kept in memory until it’s cleaned up by Ruby’s garbage collector.

As an example, let's say we'd like to uppercase all characters in a file and write it to another file. Using `File.read`, we can get the content, call `String#upcase` on the resulting string, and pass the uppercased string to `File.write`.

``` irb
irb> upcased = File.read("log/production.log").upcase
=> "I, [2018-06-27T16:45:02.843719 #9098]  INFO -- : [86A5D18C-19DD-4CBF-9D7A-461C79E98C22] STARTED GET \"/ARTICLES\" FOR 127.0.0.1 AT 2018-06-27 16:45:02 +0200\nI, [2018-06-27T16:45:02.846719 #9098]  INFO -- : [86A5D18C-19DD-4CBF-9D7A-461C79E98C22] PROCESSING BY ARTICLESCONTROLLER#INDEX AS HTML\nI, [2018-06-27T16:45:02.848212 #9098]  INFO -- : [86A5D18C-19DD-4CBF-9D7A-461C79E98C22]   RENDERING ARTICLES/INDEX.HTML.ERB WITHIN LAYOUTS/APPLICATION\nD, [2018-06-27T16:45:02.850020 #9098] DEBUG -- : [86A5D18C-19DD-4CBF-9D7A-461C79E98C22]   ARTICLE LOAD (0.3MS)  SELECT \"ARTICLES\".* FROM \"ARTICLES\"\nI, [2018-06-27T16:45:02.850901 #9098]  INFO -- : [86A5D18C-19DD-4CBF-9D7A-461C79E98C22]   RENDERED ARTICLES/INDEX.HTML.ERB WITHIN LAYOUTS/APPLICATION (1.7MS)\nI, [2018-06-27T16:45:02.851633 #9098]  INFO -- : [86A5D18C-19DD-4CBF-9D7A-461C79E98C22] COMPLETED 200 OK IN 5MS (VIEWS: 3.4MS | ACTIVERECORD: 0.3MS)\n"
irb> File.write("log/upcased.log", upcased)
=> 896
```

While that works for small files, reading the whole file into memory might be problematic when dealing with larger files. For instance, when parsing a 14-gigabyte log file, reading the whole file at once would be an expensive operation. The content of the file is kept in memory, so the app's memory footprint grows considerably. This can eventually lead to memory swapping and the OS killing the app's process.

Luckily, Ruby allows reading files line by line using `File.foreach`. Instead of reading the file's full content at once, it will execute a passed block for each line.

Its result is [enumerable](https://blog.appsignal.com/2018/05/29/ruby-magic-enumerable-and-enumerator.html), therefore it either yields a block for each line, or returns an Enumerator object if no block is passed. This enables the reading of bigger files without having to load all their content into memory at once.

``` irb
irb> File.foreach("log/production.log") { |line| p line }
"I, [2018-06-27T16:45:02.843719 #9098]  INFO -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22] Started GET \"/articles\" for 127.0.0.1 at 2018-06-27 16:45:02 +0200\n"
"I, [2018-06-27T16:45:02.846719 #9098]  INFO -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22] Processing by ArticlesController#index as HTML\n"
"I, [2018-06-27T16:45:02.848212 #9098]  INFO -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22]   Rendering articles/index.html.erb within layouts/application\n"
"D, [2018-06-27T16:45:02.850020 #9098] DEBUG -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22]   Article Load (0.3ms)  SELECT \"articles\".* FROM \"articles\"\n"
"I, [2018-06-27T16:45:02.850901 #9098]  INFO -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22]   Rendered articles/index.html.erb within layouts/application (1.7ms)\n"
"I, [2018-06-27T16:45:02.851633 #9098]  INFO -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22] Completed 200 OK in 5ms (Views: 3.4ms | ActiveRecord: 0.3ms)\n"
```

To uppercase a whole file, we read from the input file line by line, uppercase it, and append it to the output file.

``` irb
irb> File.open("upcased.log", "a") do |output|
irb*   File.foreach("production.log") { |line| output.write(line.upcase) }
irb> end
=> nil
```

So, how does reading a file line by line work without having to first read the whole file? To understand that, we’ll have to peel back some of the layers around reading files. Let's take a closer look at Ruby's `IO` class.

## I/O and Ruby's `IO` Class

Even though `File.read` and `File.foreach` exist, the [documentation for the `File` class](https://ruby-doc.org/core-2.5.1/File.html) doesn’t list them. In fact, you won’t find any of the file reading or writing methods in the `File` class documentation, because they are inherited from the parent `IO` class.

> ## I/O
> An *I/O device* is a device that transfers data to or from a computer, for example keyboards, displays and hard drives. It performs [*Input/Output*](https://en.wikipedia.org/wiki/Input/output), or *I/O*, by reading or producing streams of data. </p>
> 
> Reading and writing files from the hard drive is the most common I/O you’ll encounter. Other types of I/O include socket communication, logging output to your terminal and input from your keyboard.

The `IO` class in Ruby handles all input and output like reading and writing to files. Because reading files isn't different than reading from any other I/O stream, the `File` class directly inherits methods like `IO.read` and `IO.foreach`.

``` irb
irb> IO.foreach("log/production.log") { |line| p line }
"I, [2018-06-27T16:45:02.843719 #9098]  INFO -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22] Started GET \"/articles\" for 127.0.0.1 at 2018-06-27 16:45:02 +0200\n"
"I, [2018-06-27T16:45:02.846719 #9098]  INFO -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22] Processing by ArticlesController#index as HTML\n"
"I, [2018-06-27T16:45:02.848212 #9098]  INFO -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22]   Rendering articles/index.html.erb within layouts/application\n"
"D, [2018-06-27T16:45:02.850020 #9098] DEBUG -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22]   Article Load (0.3ms)  SELECT \"articles\".* FROM \"articles\"\n"
"I, [2018-06-27T16:45:02.850901 #9098]  INFO -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22]   Rendered articles/index.html.erb within layouts/application (1.7ms)\n"
"I, [2018-06-27T16:45:02.851633 #9098]  INFO -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22] Completed 200 OK in 5ms (Views: 3.4ms | ActiveRecord: 0.3ms)\n"
```

`File.foreach` is equivalent to `IO.foreach`, so the `IO` class version can be used to get the same result we did previously.

## Reading I/O Streams Via the Kernel

Internally, Ruby's `IO` class' reading and writing abilities are based on abstractions around [kernel system calls](https://en.wikipedia.org/wiki/System_call). The operating system's [kernel](https://en.wikipedia.org/wiki/Kernel_(operating_system)) takes care of reading from and writing to I/O devices.

### Opening Files

`IO.sysopen` opens a file by asking the kernel to put a reference to the file in the file table and creating a file descriptor in the process' file descriptor table.

> ##File Descriptors and the File Table
> [Opening](https://en.wikipedia.org/wiki/Open_(system_call)) a file returns a [file descriptor](https://en.wikipedia.org/wiki/File_descriptor) — an integer used to access the I/O resource.
>
>Each process has its own file descriptor table to keep the file descriptors in memory, and each descriptor points to an entry in the system-wide *file table*.
>
>To read from or write to an I/O resource, the process passes the file descriptor to the kernel through a system call. The kernel then accesses the file on behalf of the process, as processes don’t have access to the file table.

Opening files will *not* keep their content in memory, but the file descriptor table can get filled up, so it’s a good practice to always close files after opening them. Methods that wrap `File.open` like `File.read` do this automatically, as well as the ones taking a block.

In this example, we'll go one step further by calling the `IO.sysopen` method directly. By passing a filename, the method creates a file descriptor we can use to reference the open file later.

``` irb
irb> IO.sysopen("log/production.log")
=> 9
```

To create an `IO` instance for Ruby to read from and write to, we pass the file descriptor to `IO.new`

``` irb
irb> file_descriptor = IO.sysopen("log/production.log")
=> 9
irb> io = IO.new(file_descriptor)
=> #<IO:fd 9>
```

To close an I/O stream and remove the reference to the file from the files table, we call `IO#close` on the `IO` instance.

```
irb> io.close
=> nil
```

### Reading Bytes and Moving Cursors

`IO#sysread` reads a number of bytes from an `IO` object.

``` irb
irb> io.sysread(64)
=> " [86a5d18c-19dd-4cbf-9d7a-461c79e98c22] Started GET \"/articles\" "
```

This example uses the `IO` instance we created previously by passing the file descriptor integer to `IO.new`. It reads and returns the first 64 bytes from the file by calling `IO#sysread` with 64 as its argument.

``` irb
irb> io.sysread(64)
=> "for 127.0.0.1 at 2018-06-27 16:45:02 +0200\nI, [2018-06-27T16:45:"
```

The first time we requested bytes from the file, the cursor was moved automatically, so calling `IO#sysread` on the same instance again will produce the next 64 bytes of the file.

### Moving the Cursor

`IO.sysseek` manually moves the cursor to a location in the file.

``` irb
irb> io.sysseek(32)
=> 32
irb> io.sysread(64)
=> "9098]  INFO -- : [86a5d18c-19dd-4cbf-9d7a-461c79e98c22] Started "
irb> io.sysseek(0)
=> 0
irb> io.sysread(64)
=> " [86a5d18c-19dd-4cbf-9d7a-461c79e98c22] Started GET \"/articles\" "
```

In this example, we move to position 32, then read 64 bytes using `IO#sysread`. By calling `IO.sysseek` again with 0, we jump back to the beginning of the file, allowing us to read the first 64 bytes again.

## Reading Files Line by Line

Now, we know how the `IO` class's convenience methods open IO streams, read bytes from them and how they move the cursor's position.

Methods like `IO.foreach` and `IO#gets` can request lines line by line instead of per number of bytes. There's no performant way of looking ahead to find the next newline and take all bytes until that position, so Ruby needs to take care of splitting the file's content.

``` ruby
class MyIO
  def initialize(filename)
    fd = IO.sysopen(filename)
    @io = IO.new(fd)
  end

  def each(&block)
    line = ""

    while (c = @io.sysread(1)) != $/
      line << c
    end

    block.call(line)
    each(&block)
  rescue EOFError
    @io.close
  end
end
```

In this example implementation, the `#each` method takes bytes from the file using `IO#sysread` one at a time, until the byte is `$/`, indicating a newline. When it finds a newline, it stops taking bytes and calls the passed block with that line.

This solution works but is inefficient as it calls `IO.sysread` for every byte in the file.

### Buffering File Content

Ruby is smarter about how it does this by keeping an internal buffer of the file's content. Instead of reading the file one byte at a time, it takes 512 bytes at once and checks if there are any newlines in the returned bytes. If there are, it returns the portion before the newline and keeps the rest in memory as a buffer. If the buffer doesn't include a newline, it fetches 512 bytes more until it finds one.

``` ruby
class MyIO
  def initialize(filename)
    fd = IO.sysopen(filename)
    @io = IO.new(fd)
    @buffer = ""
  end

  def each(&block)
    @buffer << @io.sysread(512) until @buffer.include?($/)

    line, @buffer = @buffer.split($/, 2)

    block.call(line)
    each(&block)
  rescue EOFError
    @io.close
  end
end
```

In this example, the `#each` method adds bytes to an internal `@buffer` variable in chunks of 512 bytes until when the `@buffer` variable includes a newline. When that happens, it splits the buffer by the first newline. The first part is the `line`, and the second part is the new buffer.

The passed block is then called with the line and the remaining `@buffer` is kept for use in the next loop.

By buffering the file's content, the number of I/O calls is reduced while dividing the file in logical chunks.

## Streaming Files

To summarize, streaming files works by asking the operating system's kernel to open a file, then read bytes from it bit by bit. When reading a file per line in Ruby, data is taken from the file 512 bytes at a time and split up in "lines" after that.

This concludes our overview of I/O and streaming files in Ruby. We'd love to know what you thought of this article, or if you have any questions. We're always on the lookout for topics to investigate and explain, so if there's anything magical in Ruby you'd like to read about, don't hesitate to leave a comment.