Taking Mastodon's security to the next level - part 2: Exchange encrypted messages
Tags: mastodon, javascript, endtoend, showdev
Published: 2019-11-19 10:36:16

# Introduction

This is the second article in a 2-part series of blog posts that describe our endeavor to add end-to-end encryption to Mastodon: if you haven’t already, please read [Part 1: Encrypt your toots](https://dev.to/tanker/taking-mastodon-security-to-the-next-level-part-1-encrypt-your-toots-2p00) first.
In the rest of this article, we’ll refer to the Javascript code responsible for managing the UI as the *client*, and the Ruby on Rails code as the *server*.

We left on a bit of a cliffhanger - we’d managed to encrypt direct messages in the client, but hadn’t yet sent them to the server. 


Actually, sending encrypted messages to the server instead of plain text messages will lead to all sorts of interesting challenges and we’ll learn even more about Mastodon’s internals than we did in the first post.


# Adding an encrypted field in the database


Since we are encrypting only direct messages, it seems like a good idea to add an `encrypted` boolean in the database. That way, we’ll know whether statuses are encrypted or not before attempting to decrypt them.


So here’s the plan:


* The client should send an encrypted boolean to the server when calling the `api/v1/statuses` route during the composition of direct messages
* The server should store the encrypted status contents in the database, along with an `encrypted` boolean
* The server should send the encrypted text along with the `encrypted` boolean back to the client.


Let’s write a new migration and migrate the db:


```ruby
# db/migrate/20190913090225_add_encrypted_to_statuses.rb
class AddEncryptedToStatuses < ActiveRecord::Migration[5.2]
  def change
      add_column :statuses, :encrypted, :bool
  end
end
```

```console
$ rails db:setup
```

Then fix the controller:

```ruby
# app/controllers/api/v1/statuses_controller.rb
class Api::V1::StatusesController < Api::BaseController
  def create
    @status = PostStatusService.new.call(
                current_user.account,
                # ...
                encrypted: status_params[:encrypted])
  end

  def status_params
    params.permit(
       # ...
       :encrypted)
  end

end
```


Note that the controller deals only with validating the JSON request; the actual work of saving the statuses in the database is done by a service instead, so we need to patch this class as well:


```ruby
# app/services/post_status_service.rb
class PostStatusService < BaseService
# ...
  def call(account, options = {})
    @encrypted = @options[:encrypted] || false
    # …
    process_status!


  end

  def process_status!
      ApplicationRecord.transaction do
      @status = @account.statuses.create!(status_attributes)
    end
  end


  def status_attributes
    # Map attributes to a list of kwargs suitable for create!
    {
       # …
       :encrypted: @encrypted
   }.compact
  end
end
```


Let’s write a test to make sure the `PostStatus` service properly persists encrypted messages:

```ruby
# spec/services/post_status_service_spec.rb
it 'can create a new encrypted status' do
  account = Fabricate(:account)
  text = "test status update"
  status = subject.call(account, text: text, encrypted: true)
  expect(status).to be_persisted
  expect(status.text).to eq text
  expect(status.encrypted).to be_truthy
end
```


OK, it passes!


We can now use the new PostStatus API from the client code:

```javascript
// app/javascript/mastodon/actions/compose.js


export function submitCompose(routerHistory) {
  let shouldEncrypt = getState().getIn(['compose', 'shouldEncrypt'], false);
  let status = getState().getIn(['compose', 'text'], '');

  if (shouldEncrypt) {
   status = await tankerService.encrypt(status);
  }
   
  api(getState).post('/api/v1/statuses', {
    //
    status,
    encrypted: shouldEncrypt
  });
}
```


We can check that this works by composing a direct message:

![Alice sends a direct message](https://thepracticaldev.s3.amazonaws.com/i/vj87tzvvwm61r16ycfbz.png)


And then checking in the database:

```text
rails db
# select encrypted, text from statuses order by id desc;
encrypted | text
----------+---------------------------------
 t        | A4qYtb2RBWs4vTvF8Z4fpEYy402IvfMZQqBckhOaC7DLHzw…
```


Looks like it’s working as expected, so it’s time to go the other way around - sending the encrypted boolean from the server to the client.


# Displaying encrypted messages in the UI


This time we need to change the status serializer:

```ruby
# app/serializers/rest/status_serializer.rb
class REST::StatusSerializer < ActiveModel::Serializer
  attributes :id, :created_at, :in_reply_to_id, :in_reply_to_account_id,
             # ...
             :encrypted
end
```

The Javascript code that fetches the status from the Rails API does not have to change.

That being said, we still want to make it clear in the UI whether the message is encrypted or not - this is useful for debugging.

So let’s update the `StatusContent` component to display a padlock icon next to any encrypted message:

```javascript
// app/javascript/mastodon/components/status_content.js
render() {
  const encrypted = status.get('encrypted');
  let contentHtml;
  if (encrypted) {
    contentHtml = '<i class="fa fa-lock" aria-hidden="true"></i>&nbsp;' \
      + status.get('contentHtml');
  } else {
    contentHtml = status.get('contentHtml');
  }

  const content = { __html: contentHtml };
  return (
     // ...
     <div ...>
       dangerouslySetInnerHTML={content} 
     </div>
  );
}
```

![Encrypted message with padlock](https://thepracticaldev.s3.amazonaws.com/i/6txwxrnfwvsp8drivhr1.png)


Hooray, it works! We’re ready to call `decrypt` now.

# Decrypt messages


First things first, let’s patch the `TankerService` to deal with decryption:

```javascript
// app/javascript/mastodon/tanker/index.js
export default class TankerService {
  // ...

  decrypt = async (encryptedText) => {
    await this.lazyStart();

    const encryptedData = fromBase64(encryptedText);
    const clearText = await this.tanker.decrypt(encryptedData);
    return clearText;
  }
}
```


Now we’re faced with a choice. There are indeed several ways to decrypt statuses in the client code. For simplicity’s sake, we’ll patch the `processStatus` function which is called for each message returned from the server:


```javascript
// app/javascript/mastodon/actions/importer/index.js
async function processStatus(status) {
  // …
  if (status.encrypted) {
    const { id, content } = status;

    // `content` as returned by the server has a <p> around it, so
    // clean that first
    const encryptedText = content.substring(3, content.length-4);
    const clearText = await tankerService.decrypt(encryptedText);
    const clearHtml = `<p>${clearText}</p>`
    dispatch(updateStatusContent(id, clearText, clearHtml));
  }
 
}
```

Note that we call an `udpateStatusContent` action to update the status after it has been decrypted.


I won’t go through the implementation of the `updateStatusContent` action and reducers as they’re pretty standard. 


Anyway, we can check that our patch works by logging in as Alice, and then sending a message to ourselves:
  
![Alice sees her own encrypted direct message](https://thepracticaldev.s3.amazonaws.com/i/hv5crikyilsfoyy92btu.png)


# Exchanging private messages


Being able to send encrypted messages to oneself is quite impressive, but I don’t think we should stop there :)

Let’s create a new account for Bob, and look at what happens when Alice sends a message containing `@bob` - this is known as a _mention_:

![Alice to Bob](https://thepracticaldev.s3.amazonaws.com/i/spwo5debkgbevjilehh0.png)


Normally, Bob should get a notification because he was sent a direct message, but this is not the case.


Clearly there is something to fix there.


After digging into the code, here's what I found out: notifications about direct messages are generated by a class named `ProcessMentionsService`. 


Here’s the relevant part of the code:

```ruby
class ProcessMentionsService < BaseService
  def call(status)
      status.text.gsub(Account::MENTION_RE) do |match|
         mentionned_account = ...
         # …
         mentions <<  \\
           mentionned_account.mentions(...).first_or_create(states)
       end

       mentions.each { create_notification(mention) }
  end
end
```

We can see that the server looks for `@` mentions in the status text using regular expression matches and then builds a list of Mention instances.

Then something interesting happens:

```ruby
# app/services/process_mentions_services.rb
class ProcessMentionsService < BaseService
   # …
   def create_notification(mention)
    mentioned_account = mention.account

    if mentioned_account.local?
      LocalNotificationWorker.perform_async(
        mentioned_account.id, 
        mention.id, 
        mention.class.name)
    elsif mentioned_account.activitypub?
       ActivityPub::DeliveryWorker.perform_async(
         activitypub_json, 
         mention.status.account_id, 
         mentioned_account.inbox_url)
     end
  end
end
```


So the server triggers a task from the `LocalNotificationWorker` if the mentioned account is local to the instance. It turns out this will later use the websocket server we discovered in Part 1 to send a notification to the client.


Side note here: if the mentioned account is _not_ local to the instance, an Activity Pub delivery worker is involved. This is at the heart of the Mastodon mechanism: each instance can either send messages across local users, or they can use the ActivityPub protocol to send notifications across to another instance.

Back to the task at hand: it’s clear now that if the status is encrypted by the time it’s processed by the server, nothing will match and no notification will be created. That’s why Bob didn’t get any notification when we tried sending a direct message from Alice to Bob earlier.

Thus we need to process the `@` mentions client-side, then send a list of mentions next to the encrypted status to the server:


```javascript
//app/javascript/mastodon/actions/compose.js


export function submitCompose(routerHistory) {
// ...
  let mentionsSet = new Set();
  if (shouldEncrypt) {
    // Parse mentions from the status
    let regex = /@(\S+)/g;
    let match;
    while ((match = regex.exec(status)) !== null) {
      // We want the first group, without the leading '@'
      mentionsSet.add(match[1]);
    }

  const mentions = Array.from(mentionsSet);
  api(getState).post('/api/v1/statuses', {
    status,
    mentions,
    encrypted,
  });

}
```

As we did for the `encrypted` boolean, we have to allow the `mentions` key in the statuses controller and forward the `mentions` array to the `PostStatus` service:

```ruby
class Api::v1::StatusesController < Api::BaseController
  def status_params
    params.permit(
      :status,
      # ...
      :encypted,
      mentions: [])
  end


  def create
    @status = PostStatusService.new.call(
                current_user.account,                                                         
                encrypted: status_param[:encrypted],
                mentions: status_params[:mentions])
end
```



In the `PostStatus` service we forward the mentions to the `ProcessMentions` service using a `username` key in an option hash:

```ruby
# app/services/post_status_service.rb
class PostStatusService < BaseService
  def process_status!
    process_mentions_service.call(@status, { usernames: @mentions })
  end
end
```


And, finally, in the `ProcessMentions` service, we convert usernames into real accounts and create the appropriate mentions:

```ruby
# app/services/process_mentions_service.rb
class ProcessMentionsService < BaseService
  def call(status, options = {})
    if @status.encrypted?
      usernames = options[:usernames] || []
      usernames.each do |username|
        account = Account.find_by!(username: username)
        mentions << Mention.create!(status: @status, account:account)
      end
   else
     # same code as before
   end
end
```

Now we can try encrypting the following status: `@bob I have a secret message for you` and check that Bob gets the notification.

![Notification for Bob](https://thepracticaldev.s3.amazonaws.com/i/es5ki83vcjivadekopyh.png)


But when Bob tries to decrypt Alice’s message, it fails with a `resource ID not found` error message: this is because Alice never told *Tanker* that Bob had access to the encrypted message. 


For Bob to see a message encrypted by Alice, Alice must provide Bob’s public identity when encrypting the status. We still have some code to write, because in Part 1 we created and stored only private tanker identities. Luckily, the `tanker-identity` Ruby gem contains a `get_public_identity` function to convert private identities to public ones.


So the plan becomes:
* Add a helper function to access public identities from rails
* When rendering the initial-state from the server, add public identities to the serialized accounts.
* In the client code, fetch public identities of the recipients of the encrypted statuses
* Instead of calling `encrypt` with no options, call `tanker.encrypt( resource, { shareWithUsers: identities })` where `identities` is an array of public identities

Good thing we are already parsing the `@` mentions client-side :)


# Sending public identities in the initial state


First we adapt our `TankerIdentity` class so we can convert a private identity to a public one:


```ruby
# app/lib/tanker_identity.rb
def self.get_public_identity(private_identity)
  Tanker::Identity.get_public_identity(private_identity)
end
```

Then we add the `tanker_public_identity` attribute to the `User` class:

```ruby
class User < ApplicationRecord
  def tanker_public_identity
    TankerIdentity::get_public_identity tanker_identity
  end
end
```

We tell the `Account` class to delegate the `tanker_public_identity` method to the inner `user` attribute.

```ruby
# app/models/use.rb
class Account < ApplicationRecord
  delegate :email,
           :unconfirmed_email,
           :current_sign_in_ip,
           :current_sign_in_at,
           ...
           :tanker_public_identity,
           to: user,
           prefix: true
end
```

We adapt the account serializer:

```ruby
# app/serializers/rest/account_serializer.rb
class REST::AccountSerializer < ActiveModel::Serializer 
   attributes :id, :username, 
              # ...:
              :tanker_public_identity


def tanker_public_identity
    return object.user_tanker_public_identity
end
```

And now the client can access the Tanker public identities of the mentioned accounts in the initial state.


# Sharing encrypted messages

We can now collect the identities from the state and use them in the call to `tanker.encrypt()`:

```javascript
export function submitCompose(routerHistory) {
  // ...

  let identities = [];
  const knownAccounts = getState().getIn(['accounts']).toJS();
  for (const id in knownAccounts) {
    const account = knownAccounts[id];
    if (mentionsSet.has(account.username)) {
       identities.push(account.tanker_public_identity);
     }
   }
   
  // …
  const encryptedData = await tankerService.encrypt(
                                clearText, 
                                { shareWithUsers: identities });
  
  api(getState).post('/api/v1/statuses', {
  // ...
  });
}
```
  

Let’s see what happens after this code change. This time, when Bob clicks on the notification, he sees Alice's decrypted message:


![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/sj36o25jkqeitn7rdx41.png)



Done!

# What did we learn?


* We discovered how notifications are handled in Mastodon
* We found out that some server-side processing needed to be moved client-side, as is expected when client-side encryption is used.
* We implemented a fully working end-to-end encryption feature for Mastodon’s direct messages, making sure direct message can be read only by the intended recipients


If you are curious, here are some statistics about the number of changes we had to write, excluding generated files:

```text
$ git diff --stat \
   :(exclude)yarn.lock \
  :(exclude)Gemfile.lock \
  :(exclude)db/schema.rb
 41 files changed, 360 insertions(+), 40 deletions(-)
```

# Future Work


Reminder: this is a proof of concept, and many things could be improved. Here’s a list of problems and hints about their solutions.


## Improve status decryption

We are violating an implicit property of the messages in Mastodon: they are supposed to be immutable, as shown by the fact that until our patch, no action was able to change the contents of the statuses.

We probably would have to refactor the client code a bit to not violate this property, with the added benefit that the UI will no longer “flicker” when statuses go from encrypted base64 strings to clear text.

## Improving the identity verification flow

We should remove the  `@tanker/verification-ui` package and instead introduce tanker identity verification inside the existing authentication flow.

You can check out the [Starting a Tanker session](https://docs.tanker.io/core/latest/guide/start/) section of Tanker’s documentation for more details.


## Provide alternative verification methods


You may have noticed that the identity verification currently works by having Tanker and Mastodon servers holding some secrets. Also, the email provider of the users can, in theory, intercept the email containing the verification code.


If this concerns you, please note that instead of using email-based verification, we could use another verification method called the verification key. You can read more about that in the [Alternative verification methods](https://docs.tanker.io/core/latest/tutorials/alternative-verification-methods/) section of the Tanker documentation.

Please do note that in this case, users are in charge of their verification key and will not be able to access any of their encrypted resources if they lose it. 

We could implement both verification methods and let users choose between the two during onboarding.

## Implement pre-registration sharing

The code assumes all users sending or receiving direct messages already have a Tanker identity registered. This can also be solved by using a Tanker feature called [Pre-registration sharing](https://docs.tanker.io/core/latest/tutorials/pre-registration-sharing/).


## Make encryption work across instances

Finally, our implementation works only if the sender and receiver of the direct messages are on the same instance - we need to make encryption work with the ActivityPub protocol.

I have a few ideas but fixing it seems non-trivial. Still, it would make for a pretty nice challenge :)


# Conclusion

Thanks for reading this far. Writing the patch was a nice experience: Mastodon’s source code is clean and well-organized. You can browse the changes on the [pull request on GitHub](https://github.com/tootsuite/mastodon/pull/12428).


I hope this gave you an idea of the possibilities offered by Tanker. If you’d like to use Tanker in your own application, please [get in touch with us](https://tanker.io/).


Feel free to leave a comment below and give us your feedback!