How I migrated from Flask-Security(-Too) to Devise
Tags: rails, flask, webdev
Published: 2023-12-07 11:07:49

Recently I started a rewrite of one of my smallish projects (in terms of complexity, but with ~200 users, so I must handle that with care) from Flask to Ruby on Rails.
Why? I'm planning on writing another article about that, so I don't want to go into details here, but in short, I believe it will enable me to write the app both faster and better. But there are some hurdles to overcome to get there.

For some time, I ignored the authentication part of the app. So today, I made my first attempt at making that work.

In Flask, I used the `Flask-Security-Too` package from the start. I never tried to implement any authentication logic myself (as you probably never should for, you know, security reasons).
I knew that in Rails, I could use the `devise` gem.
But it means jumping from one black box into another. I knew that it might bring some difficulties.

> Warning: I'm no expert on security by any means! I understand really little about this, just enough to make it work. There might be some security concerns I'm not aware of. Read this just as a story, not a tutorial!

There was some little work involved in correctly adding columns to the database to support `devise` (as some column names are the same in both libraries and some are not), but not that much. The most annoying is that I had a `password` column, and that caused some problems with Rails/devise. But after some tries with migrations, I got sorted that out.

The main problem, as expected, was being able to validate the same hashes in Rails that `Flask-Security` created automatically for me.

# The Problem - Password hashes

Short aside if this is new for you: what are password hashes?
It would be very unwise to save passwords to the database as-is. I certainly don't want to have that responsibility on me.
So instead password hashes are usually stored in the database, and on login attempt hash is calculated from the provided password and compared to the saved one.
So the app knows the same password was provided as when the user registered, but is none the wiser about the actual value.

_When I looked into my database, I had hashes like `$2a$12$etyAllfaDgIDC61ZUS9UB.peoAIYpZborbU4T/6c8SejvrIhbIL46` there._

Ok, back on track. The trouble is that there are many different ways to hash passwords.
You can use different algorithms (sha256, bcrypt, argon,..), they can have different configurations, there may be some salt or pepper involved,..

There's basically NO CHANCE that a different library (`devise` in my case) will hash, and therefore validate, passwords the same way as some other (`flask-security` for me).

So it was no surprise that after adding `devise` to my project, logging in with my credentials wasn't working.

The easy way out for me would be the following - just toss out all passwords and send mail to my users that they need to use the `Forgotten password` feature. Well, not gonna happen, thank you very much.

So I needed to find out way to migrate successfully.

There were two parts to the solution:
1) Slightly changing mechanism for Devise-provided password-validation to validate using its built-in mechanism first, and if that fails, try our own validator, simulating how `Flask-Security` does validation.
2) Writing the validator

## Allowing both Devise and Flask-Security-like validation
The first part is quite straightforward:

```ruby
# app/models/user.rb
class User < ApplicationRecord
  # Include default devise modules. Others available are:
  # :confirmable, :lockable, :timeoutable, :trackable and :omniauthable
  devise :database_authenticatable, :registerable,
         :recoverable, :rememberable, :validatable

  (...)

  def valid_password?(password)
    if Devise::Encryptor.compare(self.class, encrypted_password, password)
      true
    elsif flask_security_compare(password)
      true
    else
      false
    end
  end

  (...)
end
```

It's probably pretty self-explanatory, I'm just overriding the default `valid_password?` method Devise is providing, adding another way to validate the password. Nice, let's move into the second part - how will `flask_security_compare` work?

## Simulating Flask-Security hashing
Now this was a bit more difficult (took me a few hours to solve).

First, I needed to understand what Flask-Security is doing when hashing passwords.

Luckily, I was already using `bcrypt` as my hashing algorithm (as it's the default for `Flask-Security`), the same as Devise uses.
I needed to check that it is configured the same way, mainly the number of stretches (technique making hashing slower, thus making brute-force attacks more difficult).

Sidenote:
Remember the hash I showed you before? Here it is again `$2a$12$etyAllfaDgIDC61ZUS9UB.peoAIYpZborbU4T/6c8SejvrIhbIL46`._
`$2a` tells us it's bcrypt using 2a variation
`$12` is the number of stretches

Now for a tricky part.

When the password is hashed using `bcrypt`, there's `salt` provided. That's a way to make pre-calculating hashes nearly impossible (preventing usage of the infamous rainbow tables).

Where does this salt come from?

It's generated randomly.

For a moment I was afraid that it meant I would be unable to simulate this - how can I recreate something random? Of course, that's not the case - bcrypt needs to keep this value around. And it does so directly in the hash string:

in `$2a$12$etyAllfaDgIDC61ZUS9UB.peoAIYpZborbU4T/6c8SejvrIhbIL46`, `etyAllfaDgIDC61ZUS9UB.` is the salt (22 characters after the initial part).
Only the rest (`peoAIYpZborbU4T/6c8SejvrIhbIL46`) is the actual hash!

Nice! And even nicer - I don't need to parse the value manually, I can use the library for that (as suggested by Copilot):
```ruby
legacy_password_hash = "$2a$12$etyAllfaDgIDC61ZUS9UB.peoAIYpZborbU4T/6c8SejvrIhbIL46"
bcrypt_password_hash = BCrypt::Password.new(legacy_password_hash)
puts bcrypt_password_hash.salt
# $2a$12$etyAllfaDgIDC61ZUS9UB.
```

Ok, we have a salt, let's hash some passwords!

```ruby
hashed_password = ::BCrypt::Engine.hash_secret(password, bcrypt_password_hash.salt)
```

Well, I did get a result, but it was different than what I had in the database...

The reason is that `Flask-Security` uses one more security enhancement.

It doesn't use the password directly for hashing but encodes it to HMAC using a server-wide secret key (set as `SECURITY_PASSWORD_SALT` env variable).

I had to look into `Flask-Security` [source code](https://github.com/Flask-Middleware/flask-security/blob/1c8062dd6acda6e1e8928df8a991061a705a8d17/flask_security/utils.py#L307) to find how exactly it's done:

```python
def get_hmac(password: SB) -> bytes:
    """Returns a Base64 encoded HMAC+SHA512 of the password signed with
    the salt specified by *SECURITY_PASSWORD_SALT*.

    :param password: The password to sign
    """
    salt = config_value("PASSWORD_SALT")

    if salt is None:
        raise RuntimeError(
            "The configuration value `SECURITY_PASSWORD_SALT` must "
            "not be None when the value of `SECURITY_PASSWORD_HASH` is "
            'set to "%s"' % config_value("PASSWORD_HASH")
        )

    h = hmac.new(encode_string(salt), encode_string(password), hashlib.sha512)
    return base64.b64encode(h.digest())
```

and then recreated the algorithm in ruby:


```ruby
def get_hmac(password)
    require 'openssl'
    require 'base64'

    salt = ENV['LEGACY_SECURITY_SALT']

    if salt.nil?
      raise "The configuration value `LEGACY_SECURITY_SALT` must not be None when the value of `SECURITY_PASSWORD_HASH` is set to #{ENV['SECURITY_PASSWORD_HASH']}"
    end

    digest = OpenSSL::Digest.new('sha512')
    hmac = OpenSSL::HMAC.digest(digest, salt, password)
    Base64.encode64(hmac)
  end
```

and used that to write the `flask_security_compare` function I need:

```ruby
def flask_security_compare(password)
    bcrypt_password = BCrypt::Password.new(legacy_password_hash)

    hmaced_password = User.get_hmac(password)
    hashed_password = ::BCrypt::Engine.hash_secret(hmaced_password, bcrypt_password.salt)

    Devise.secure_compare(hashed_password, encrypted_password)
end
```

And it still didn't work.
I was a little desperate by then, but after some debugging and comparing of strings I found out small problem:
The ruby implementation sometimes added `\n` symbols into the `hmaced_password`.
Fast fix is this: `Base64.encode64(hmac).chomp.gsub(/\n/, '')`.

Now I'm just curious if this works with every password there is in my database. Let's hope.
And maybe show a banner to users warning them that it might happen.

Oh, I forgot one more improvement - if the user logs in using the "old mechanism", it can rehash the password to update the hash. This way, I can remove the `flask` branch later (like in a year or so). I already did something similar when I was migrating from SHA256 to the bcrypt algorithm in my older project.

Now I'm done.

So, I learned a bit about how password hashing works in `Flask-Security`, which was interesting. And I'm really happy that I can continue with my rewrite to Rails, knowing that users will be able to log in using their old passwords.

> Again, this is not a tutorial. Please, if you do some authentication stuff, be careful. And if you find any mistakes here, let me know in the comments!

Update:
I realized I didn't address properly one potential downside to this. As password validation tries two way of hashing, it makes guessing password easier, as there are potentially two strings that produce the desired hash. That's not great, as it's lowering security of our authentication, but as we use otherwise safe way, it shouldn't do much damage (half the time for brute-forcing will still be a lot). We could migitage this by having `flask_password_hash` and `password_hash` as different fields, and only allowing one of them to be present.