Looking Inside a Ruby Gem
Tags: ruby, rails, tutorial
Published: 2020-04-21 21:09:29

---
title: Looking Inside a Ruby Gem
published: true
description: Have you ever wondered what a gem is? What's inside a Ruby gem? Having written our gemspec, it's only natural to use it to learn how a Ruby package is built.
tags: ruby,rails,tutorial
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/enicdrzbtyyp2ypoh34e.jpeg
canonical_url: https://piotrmurach.com/articles/looking-inside-a-ruby-gem/
---

If you have used Ruby for any length of time you have probably, knowingly or not, used a gem. It could've been already installed on your system as part of your Ruby release. You may also have downloaded it with the RubyGems package manager. Irrespective of how you got hold of a gem, you required it as a dependency in your file and used it. But, have you ever wondered what a gem is? What's inside a Ruby gem? Having written a Ruby gem manifest file in the <a href="https://dev.to/piotrmurach/writing-a-ruby-gem-specification-5e4g">Writing a Ruby Gem Specification</a> article, it's only natural to use it to learn how a Ruby package is built.

In this article, I'd like to take you to the RubyGems factory to see where it all begins. We will first learn how to create a Ruby gem and then unpack it and look inside the internals. Whether you intend to share your own gems or continue using third-party ones, knowing the inner workings of a gem will be useful. If anything it will make you appreciate the elegance with which the whole Ruby ecosystem works.

Now, on to the fun.

# What is a Gem?

Any popular programming language has an ecosystem of packages that solve various problems. These packages capture reusable functionality and save tons of time for everyone else. To be accessible to everyone, libraries need a central storage place. A service for automated uploading and downloading of named packages referred to as - a registry.

Examples of popular registries are NPM (<a href="https://www.npmjs.com/" taret="_blank">www.npmjs.com</a>) for JavaScript, Hex (<a href="https://hex.pm/" target="_blank">hex.pm</a>) for Elixir and Erlang code, and Crates (<a href="https://crates.io/" target="_blank">crates.io</a>) for Rust. Ruby also has an official place to store and distribute its packages: the <a href="https://rubygems.org/" target="_blank">rubygems.org</a>. This is a service that provides you with the ability to browse and download hundreds of thousands of Ruby packages called a gem.

Each gem in a registry has a unique name and a release number. The Ruby community has a taste for quirky and whimsical names. Don't believe me, look at these:

* <a href="https://rubygems.org/gems/cucumber" target="_blank">Cucumber</a> - tool for running automated tests written in plain language.
* <a href="https://rubygems.org/gems/jekyll" target="_blank">Jekyll</a> - a static site generator.
* <a href="https://rubygems.org/gems/nokogiri" target="_blank">Nokogiri</a> - HTML and XML parser.
* <a href="https://rubygems.org/gems/sinatra" target="_blank">Sinatra</a> - a minimal web framework.
* <a href="https://rubygems.org/gems/unicorn" target="_blank">Unicorn</a> - a web server.

The vibrant and large ecosystem of Ruby gems plays a considerable role in making the Ruby language as popular as it is now. To be part of this success and to create our own gems, we need to learn about the tool that makes it all possible, the RubyGems package manager.

## The RubyGems Package Manager

When a Ruby language is installed on your system it includes a handful of gems. One of these gems is the **RubyGems** gem (this is a bit meta, a gem to manage gems). The RubyGems is a powerful package manager that will do things for you like:

* Installation of third-party gems.
* Dependencies resolution.
* Searching local and remote packages.
* Inspection of gems including their files and metadata.
* Building and serving `RDoc` documentation.
* Creation and publishing of gems.

This is not an exhaustive list but it should give you a taste of how many tasks a full-blown package manager handles. As of this writing, Ruby 2.7 installs RubyGems 3.1.2 on my system. The RubyGems gem provides you with a command-line tool that will support all the before listed tasks called the **gem**. I agree this is a bit confusing. Both a Ruby package and the tool to manage these packages are named the same.

You may also have used a tool called Bundler to install gems. Both Bundler and RubyGems are partners in the same crime. They help you above all manage gem dependencies. Bundler's focus is on dealing with the installation of gems for a local application. In contrast, RubyGems manages all the gems installed on your system under a given Ruby version. The plot thickens even more starting from Ruby version 2.6. In this version, the Bundler has been merged into the RubyGems as a default gem alongside packages like *irb*, *logger* or *fileutils*.

For us, the important feature of RubyGems is that it can convert a gem specification into a package. That's what we're going to take a closer look at next.

## Building a Gem

Most likely you used the **gem** command to install or uninstall a dependency. There are more than 30 commands available at your fingertips. In particular, the **gem build** command is what we're going to look at. Its job is to take a manifest file and turn a bunch of files into a Ruby gem. The typical layout of a gem follows a well-established pattern for organising files. For example, the layout of the **emoticon** gem that we used to write a manifest file for looks like this:

```
emoticon/
├── exe
│   └── emote
├── lib
│   ├── emoticon
│   │   └── version.rb
│   └── emoticon.rb
├── CHANGELOG.md
├── LICENSE.txt
├── README.md
└── emoticon.gemspec
```

For our demonstration, we're going to use the already created **emoticon.gemspec** that contains all the information necessary to bake a new gem.

To create a gem from our gemspec, run the `gem build` command:

```bash
$ gem build emoticon.gemspec
```

This will display confirmation information in the console:

```
Successfully built RubyGem
Name: emoticon
Version: 0.1.0
File: emoticon-0.1.0.gem
```

At this point, you will have the **emoticon-0.1.0.gem** file in the same directory. Nice!

Under the covers the **build** command delegates its work to the `Gem::Package` class and the class level `build` method. This method accepts as an argument `Gem::Specification` instance and few options. That's exactly the instance we used to describe metadata about the emoticon gem in our manifest file. As a reminder here is the *emoticon.gemspec* snippet, for the sake of clarity:

```ruby
# frozen_string_literal: true

require_relative "lib/emoticon/version"

gemspec = Gem::Specification.new do |spec|
  spec.name        = "emoticon"
  spec.version     = Emoticon::VERSION
  ...
end
```

Instead of relying on the `gem build` command, we will create a package manually to learn what's involved. To begin, we create a new file *build.rb* and load the `"rubygems/package"` dependency. As it happens, all we need to do to build a gem is add a couple lines of code. First, we load the gem specification. Next, we invoke the `Gem::Package`'s `build` method with a gemspec as an argument:

```ruby
require "rubygems/package"

gemspec = Gem::Specification.load("emoticon.gemspec")
Gem::Package.build(gemspec)
```

With these two lines in place, we can now execute our *build.rb* file:

```bash
$ ruby build.rb
```

This produces exactly the same result as before - an **emoticon-0.1.0.gem** file. Now, we have a gem for further exploration.

## What's Inside a Gem?

The `emoticon-0.1.0.gem` has an unusual **.gem** extension name. But, this file is nothing more than an archived directory packaged using the <a href="https://en.wikipedia.org/wiki/Tar_(computing)" target="_blank">tar</a> Unix utility. We can take a look inside of the archive and list its content to the console with the tar utility. All we need to do is pass the *-t* flag for printing to the terminal and *-f* option for reading from a file:

```bash
$ tar -tf emoticon-0.1.0.gem
```

Inside the tar archive file, we find a set of three gzipped files:

```
checksums.yaml.gz
data.tar.gz
metadata.gz
```

If you build your gem and sign it with a cryptographic key then the archive will have a total of six files. Each gzipped file will have a matching signature file:

```
checksums.yaml.gz
checksums.yaml.gz.sig
data.tar.gz
data.tar.gz.sig
metadata.gz
metadata.gz.sig
```

What are these signature files?

To create a signature file, the RubyGems selects a digest algorithm from the standard library:

```ruby
# lib/rubygems/security.rb

module Gem::Security
 DIGEST_ALGORITHM =
    if defined?(OpenSSL::Digest::SHA256)
      OpenSSL::Digest::SHA256
    elsif defined?(OpenSSL::Digest::SHA1)
      OpenSSL::Digest::SHA1
    else
      require 'digest'
      Digest::SHA512
    end
end
```

In the first step, the digest algorithm is applied to every file in the archive to create a summary of its content. The summary is a unique value of a fixed-length often referred to as a file checksum or digest. Then, a signing key and a certificate are combined into a signing instance of `Gem::Security::Signer`. The signer signs the file's digest and creates a digital signature that is then saved to a file with *.sig* extension. To see the content of the signature, a base64 encoded binary file, we can use the *base64* command:

```bash
$ base64 data.tar.gz.sig
```

In this case, we see some random characters in the terminal output, truncated to save the space:

```
bAclqtMmrTKFSMFn74w8kwjRrmv/8Wlc1prNXxHfYtqHypaFdnoIOKKPhYAp3D5MKiqmqIsY8VTb8gQvbT/rEcMm40C57fh5JEQ3I7Tkvs0D76nL+cIaqnnqJ1H5Irknhkj+fu...
```

The signature can be thought of as a software equivalent of what we do in everyday life when we sign a paper document with a pen. By signing a document we verify its accuracy and authenticity. In our case, the signature files provide an extra layer of protection when installing gems in two ways. First, they allow us to verify that the gem source files haven't changed since they were originally signed. Second, they ensure that the signature belongs to the gem's author who owns the private signing key.

By default gems are installed as unsigned without checking their file signatures. This leaves the possibility open that a gem may have been tampered with by an attacker. To take advantage of the signatures during the gem's installation process, you need to use the *- -trust-policy* option or short flag *-P* with the highest and most secure policy:

```bash
$ gem install emoticon -P HighSecurity
```

With the signatures files out of the way, we can come back to our checksums, data and metadata compressed files. Our good friend the `Gem::Package` has more to teach us about how it performs its packaging process. Diving into the `build` method, we discover that all these three archived files are generated using the `Gem::Package::TarWriter` class. The tar writer does its work by exposing a few helper methods via an internal DSL (Domain Specific Language) that add content to the main gem archive file:

```ruby
# lib/rubygems/package.rb

@gem.with_write_io do |gem_io|
  Gem::Package::TarWriter.new gem_io do |gem|
    add_metadata gem
    add_contents gem
    add_checksums gem
  end
end
```

Let's go ahead and extract all the files from the emoticon gem archive:

```bash
$ tar -xf emoticon-0.1.0.gem
```

One by one, we're going to examine files from the emoticon gem archive so we can learn their purpose.

## Inspecting the Files

The first file that we're going to take under a microscope is a compressed `YAML` file called *checksums.yaml.gz*. The file compression is done with a lesser-known **zlib** gem from the Ruby standard library. This gem provides an interface to general purpose and portable *zlib* utility for compressing files. Among the formats it supports is the **gzip** (**.gz**) format.

In more detail, the `Gem::Package` offloads compression of all the archived files to the `gzip_to` method. This method accepts as an argument a file content represented as IO stream. Then it uses the `Zlib::GzipWriter` to turn the stream into a gzipped version:

```ruby
# lib/rubygems/package.rb

def gzip_to(io)
  gz_io = Zlib::GzipWriter.new io, Zlib::BEST_COMPRESSION
  gz_io.mtime = @build_time
  yield gz_io
ensure
  gz_io.close
end
```

Since the *checksums.yaml.gz* is a binary file we need to do something to view its content. Fortunately, we don't have to unpack it to look inside, as we can use the *gzcat* utility:

```bash
$ gzcat checksums.yaml.gz
```

What we see inside at the top level are the keys that correspond with the names of the digest algorithms used. In our case, the sha256 and sha512 algorithms. Underneath each algorithm name are pairs of gzipped file name and its checksum. The checksums are digital summaries of the file content generated with the particular algorithm. For example, the file checksums under the sha256 key are represented as a string of 64 hexadecimal encoded characters. Thus it all looks something like this:

```yaml
---
SHA256:
  metadata.gz: bdac75ac0dab99ec3cdb10692ef70f8b0966ef0aff7fb2c40b3c0ebf0440667d
  data.tar.gz: f0004274a83869c47e487eeb06c655b13c14601350eb8f6f4299ea6e0259b2d4
SHA512:
  metadata.gz: 03d8c338cee67683cc5e032a4e54153c9a73496e8960173a60b74b31fbe372e8786b86979efd6a923e0842dc132b2d4bdf020eff299b13a970018a31b5503325
  data.tar.gz: c5c52e2f82d90cc9d249b2dcc3500b3e29216f5db4049557b365f5051f34df19e5152519793b60b9691e1b951ec54fb6fa8e6c39ef673522cc1e19d72bb01bbf
```

The purpose of the checksums file is to help verify the integrity of the *data.tar.gz* and *metadata.gz* files during the gem installation process.

The next file we're going to zoom in on is the *data.tar.gz*. Since this is a gzipped tar archive, to see inside we use the familiar technique:

```bash
$ tar -ztf data.tar.gz
```

This file contains the source code files that were listed in the gem specification. Here is the list from emoticon gem:

```
CHANGELOG.md
LICENSE.txt
README.md
exe/emote
lib/emoticon.rb
lib/emoticon/version.rb
```

The final *metadata.gz* file is a `YAML` archive that includes all the metadata extracted from the gem's manifest file. If we look inside the file:

```bash
$ gzcat metadata.gz
```

We're greeted with a long output akin to the following truncated version:

```yaml
--- !ruby/object:Gem::Specification
name: emoticon
version: !ruby/object:Gem::Version
  version: 0.1.0
platform: ruby
authors:
- Piotr Murach
autorequire: 
bindir: exe
cert_chain: []
date: 2020-03-01 00:00:00.000000000 Z
dependencies:
- !ruby/object:Gem::Dependency
  name: tomlrb
  requirement: !ruby/object:Gem::Requirement
    requirements:
    - - "~>"
      - !ruby/object:Gem::Version
        version: '1.2'
  type: :runtime
  prerelease: false
  version_requirements: !ruby/object:Gem::Requirement
    requirements:
    - - "~>"
      - !ruby/object:Gem::Version
        version: '1.2'
...
```

What's interesting here are all the `!ruby/object` notations. What are these? The `YAML` format allows us to serialize simple values like strings, arrays or hashes. But it also can serialize pretty much any Ruby object. By default, `YAML` will serialize all the object's instance variables and their values. For example, if we create a `Gemspec` class from a `Struct` object with some attributes:

```ruby
require "yaml"
Gemspec = Struct.new(:name, :date, :version)
gemspec = Gemspec.new("emoticon", Time.now, "0.1.0")
```

And then print the object content in a `YAML` format to the console:

```ruby
puts YAML.dump(gemspec)
```

We will see all the instance variables serialized:

```yaml
--- !ruby/struct:Gemspec
name: emoticon
date: 2020-04-10 14:00:41.077868000 +01:00
version: 0.1.0
```

Yet, we're not stuck with only serializing object's attributes. We can control what attributes `YAML` is going to use with the `to_yaml_properties` method. For example, this is how the `Gem::Requirement` specifies it's `YAML` transformation under the hood:

```ruby
# lib/rubygems/requirement.rb

class Gem::Requirement
  def to_yaml_properties # :nodoc:
    ["@requirements"]
  end
end
```

In our case, the entire `Gem::Specification` is serialized this way. All its attributes that may point to other objects are recursively serialized and saved in the `YAML` file.

If you wish to take a look at the specification of any gem installed on your system, you can run the *gem spec* command:

```bash
$ gem spec package-name
```

This will print all the gem's metadata information in your console.

You should also notice that the gem specification file is only needed when building a gem. It shouldn't be part of your release. The metadata is extracted to a `YAML` file and that's what is used during a gem installation.

## Conclusion

You may have thought of gems as nondescript blobs of functionality that somehow appear on your system under some weird and wonderful name. My hope is that this article clarified the structure of a gem and how it accomplishes its functionality in the Ruby ecosystem. 

After all, we learnt that there is nothing new under the sun. The old and trusted Unix tools for archiving and compression serve as the foundation for packaging gems. We experienced how the succinct YAML format keeps the data readable and machine friendly. We saw how the OpenSSL toolkit adds a security layer to make our gems resilient and dependable.

I don't know about you but I like to know the nitty-gritty details of the tools I use. I'm in awe and have nothing but great admiration for the developers who provide such solutions. It takes an ingenious effort to turn the concept of reusable code into a practical solution that scales and makes millions of Ruby projects possible!

Thanks for taking the time to read the article and I hope you found it useful.

Stay hungry and creative!

---

This article was originally published on <a href="https://piotrmurach.com/articles/looking-inside-a-ruby-gem/" target="_blank">PiotrMurach.com</a>.

Photo by <a href="https://unsplash.com/@jdphotomatography">Jason D</a> on Unsplash.