Rocket Tutorial 04: Data Persistency and Rocket (with MongoDB)
Tags: rust, mongodb, tutorial, beginners
Published: 2020-11-30 13:24:57

[<span>Photo by <a href="https://unsplash.com/@lu_pl_ph?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Luigi Pozzoli</a> on <a href="https://unsplash.com/s/photos/rock-climbing?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></span>, modified (cropped)]

We will continue to build a CRUD API with Rocket. 
This installment: persistency with a DB!

The code for this tutorial can be found in this repository: [github.com/davidedelpapa/rocket-tut](https://github.com/davidedelpapa/rocket-tut), and has been tagged for your convenience:

```bash
git clone https://github.com/davidedelpapa/rocket-tut.git
cd rocket-tut
git checkout tags/tut4
```

## We need to secure the data.

Right now we have everything saved in memory. This is not an ideal situation, because we need persistency. Even if we could make sure that the instance of the server will never fail (which is impossible), the first time we would need to shut the server down, all data would be gone.

We can easily test for this (yes TDD, Test Driven Development, is a thing! Although I don't use it too often :-) ). Le't create a new *tests/persistency_test.rs*:

```rust
use rocket::local::Client;
use rocket_tut::rocket_builder;
use rocket::http::{ContentType, Status};
use rocket_tut::data::db::ResponseUser;
use serde_json;

#[test]
fn create_and_persist_test(){
    // We make sure that client1 gets properly disposed of
    {
        let client1 = Client::new(rocket_builder()).expect("Valid Rocket instance");
        let mut response = client1.post("/api/users")
            .header(ContentType::JSON)
            .body(r##"{
                "name": "John Doe",
                "email": "jd@m.com",
                "password": "123456"
            }"##)
            .dispatch();
        assert_eq!(response.status(), Status::Ok);
        assert_eq!(response.content_type(), Some(ContentType::JSON));
        let response_body = response.body_string().expect("Response Body");
        let user: ResponseUser = serde_json::from_str(&response_body.as_str()).expect("Valid User Response");
        assert_eq!(user.name, "John Doe");
        assert_eq!(user.email, "jd@m.com");
    }

    // Let's create a new client and ask for info there using the email
    let client2 = Client::new(rocket_builder()).expect("Valid Rocket instance");
    let mut response = client2.get(format!("/api/users/{}", "jd@m.com")).dispatch();
    let response_body = response.body_string().expect("Response Body");
    let user: ResponseUser = serde_json::from_str(&response_body.as_str()).expect("Valid User Response");
    assert_eq!(response.status(), Status::Ok);
    assert_eq!(response.content_type(), Some(ContentType::JSON));
    assert_eq!(user.name, "John Doe");
    assert_eq!(user.email, "jd@m.com");
}
```

We can now test this:

```bash
cargo test -- --nocapture create_and_persist_test

[.. skip ... skip ..]

thread 'create_and_persist_test' panicked at 'Valid User Response: Error("invalid type: string \"user j.doe@m.com not found\", expected struct ResponseUser", line: 1, column: 28)', tests/persistency_test.rs:31:76
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test create_and_persist_test ... FAILED
```

(we cut much of the output to focus on the problem)

The test failed because the API could not find the user: `invalid type: string \"user j.doe@m.com not found\"`.
Thus, *serde_json* could not construct a valid `ResponseUser` and it failed.

Since we did not use the same server, but we re-created it using the same rocket constructor, the new server did not have the data that indeed we sent to the first one.

Well this is a problem, because in real life failure *does* happen, and recovery must be a feature (besides scaling: I mean, if two instances of the server cannot communicate with one another, at least by passing data to each other... what a mess!)

## MongoDB to the rescue

We need a way to store persistently rocket's state. Enters DBMS: a DataBase management System. 

If you have it not installed on your system, you can use an online service ([Atlas](https://www.mongodb.com/cloud/atlas), for example. There's a free tier to start testing it). 
Otherwise:

```bash
# in the following substitute "focal" in "ubuntu focal" with "bionic" or "xenial"...
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list
sudo apt-get update
sudo apt-get install -y mongodb-org
```

(Please, check the [official installation page](https://docs.mongodb.com/manual/administration/install-community/) for other systems).

Now we need to start it.

```bash
sudo systemctl start mongod
```

Once everything is in order, let's start to use it in our code:

```bash
cargo add r2d2 r2d2-mongodb dotenv
```

We added few crates:

[dotenv](https://crates.io/dotenv) makes it easier to use secrets in our project.

[r2d2](https://crates.io/r2d2) [r2d2-mongodb](https://crates.io/r2d2-mongodb) are needed to pool the `mongodb`, that is, to create a pool of connections in order to speed up the communications

As database, we are using [MongoDB](https://www.mongodb.com), which is maybe the most known noSQL database. Notice we are not using the current crate ([mongodb](https://crates.io/mongodb)), but that which is exposed in `r2d2-mongodb`. More on that later. 

This database is not relational: it is not a series of tables, but instead a series of collections of documents. Such documents can be also different one from the other: that is, they need not to be homogeneous. 

In practice this means that if we add or vary some fields on Users in the future, the old users can remain the same (we need still to provide support for them, though),  while the new one can use the new features. Also, if we have different kind of users, that need different fields, we do not need to keep them in two different tables, or add all fields to all users, as in a relational DB, but we could have different kinds of users in the same collection at the same time.

In sum, sometimes a noSQL DB is more fit to the purpose, as in this case, so we'll take advantage of this.

Besides, MongoDB uses a format for documents, BSON, which is compiled (binary) JSON: changing from one to the other is a matter of milliseconds, so we can say that if you are already dealing with JSON documents, to store them in MongoDB is a naural choice.

At last, a CAVEAT: using `mongodb` in Rust means matching out for a lot of `Result`, `Option`, and `Result<Option>`, because the connection to the DB is usually over the internet (MongoDB has its own database-as-a-service as well). As we know, net connection is more prone to fail than, say, a connection over a local socket to an instance of MySQL running on the same machine as the server. That is why the unwrapping of the responses can take looooot of code.

That said, there are two ways to go about to check for many `Result`'s and `Option`'s: either `match` everything, or using *combinators*: `ok()`, `unwrap_or()`, `and_then()`, etc, etc... I prefer to `match` and follow strict indentation, so that it is clearer at the end what we are matching for. We use indentation to help understanding the patterns of `Ok(_)`/`Err(_)` and `Some(_)`/`None`.

Maybe it's matters of style: being also proficient in Python means I can reason about code using indentation much easier than follow a pattern of `dot-combinator` (`.and_then()`, `.ok()` etc...). Those who prefer combinators will forgive me, I hope.

### A note on mongodb and rocket_contib

Before the last release there was a direct support of *mongodb* in the databases section of *rocket_contrib*. Now the support has been removed. I tried even with an older version, but I could not manage to let the two connect.

It is a pity because there is a good support for the diesel ecosystem instead, which is geared towards relational databases (MySQL, Postgres, SQLite), but the support of the NoSQL databases is heavily lacking... Anyway, let's start coding our way around in MondoDB.

## Implementing it in code

Let's first modify *src/data/mod.rs*:

```rust
pub mod db;
pub mod mongo_connection;
```

Now we need of course a *src/data/mongo_connection.rs*:

```rust
use std::ops::Deref;
use std::env;
use dotenv::dotenv;
use r2d2::PooledConnection;
use r2d2_mongodb::{ConnectionOptions, MongodbConnectionManager};
use rocket::http::Status;
use rocket::request::{self, FromRequest};
use rocket::{Outcome, Request, State};
```

First thing to notice: `dotenv` has to be used in connection with `std::env`. 
The method it uses is as follows: `dotenv` reads the variables from a `.env` file, and assigns them to the environment variables. Then, we can use `std::env` in order to retrieve them.

We are using `std::ops::Deref` because we will need to impl the connection to the mongodb pool later on.

We declare two types for later use:

```rust
type Pool = r2d2::Pool<MongodbConnectionManager>;
type PooledConn = PooledConnection<MongodbConnectionManager>;
```

The type `Pool` refers to the pooled database, and the `PooledConn` to the pooled connection to that DB.

```rust
pub struct Conn(pub ConnType);
```

That is the connection we will use.

```rust
impl<'a, 'r> FromRequest<'a, 'r> for Conn {
    type Error = ();

    fn from_request(request: &'a Request<'r>) -> request::Outcome<Conn, ()> {
        let pool = request.guard::<State<Pool>>()?;
        match pool.get() {
            Ok(database) => Outcome::Success(Conn(database)),
            Err(_) => Outcome::Failure((Status::ServiceUnavailable, ())),
        }
    }
}
impl Deref for Conn {
    type Target = PooledConn;
    fn deref(&self) -> &Self::Target {
        &self.0
    }
}
```

We impl for our Conn a `FromRequest` so that Rocket can use it as a guard, and `Deref`, so that we can access the underlying `MongodbConnectionManager` and transfer it to the `PooledConn`.

Now, let's create a `init_pool()` function to fire up the pooled connection

```rust
pub fn init_pool() -> Pool {
    dotenv().ok();
    let mongodb_address = env::var("MONGODB_ADDRESS").expect("MONGODB_ADDRESS missing");
    let mongodb_port = env::var("MONGODB_PORT").expect("MONGODB_PORT missing");
    let database = env::var("MONGODB_DATABASE").expect("MONGODB_DATABASE missing");
    //let mongodb_user = env::var("MONGODB_USER").expect("MONGODB_USER missing");
    //let mongodb_password = env::var("MONGODB_PASSWORD").expect("MONGODB_PASSWORD missing");
```

`dotenv.ok()` is all we need to transfer the variables from our `.env` file to the environment.

Then we retrieve each variable with `env::var()`. We use `expect()` here, because in this way the server will panic at  start up, sending us the message with which variable was not found.

Notice two things: the part that is commented out is relative to the MongoDB user and password: in a local development they are not needed, but they will be necessary if we use the Atlas service, for example, or if we want to secure it for online access (spinning up our server in a virtual server somewhere, and the MongoDB instance in another server, elsewhere, as is good practice).

Second thing: the `.env` file! (to be kept possibly at the root of the project)

```env
MONGODB_ADDRESS=127.0.0.1
MONGODB_PORT=27017
MONGODB_DATABASE=users
MONGODB_USER=me
MONGODB_PASSWORD=password
```

This is just an example, for use in a local setting. However, it is good practice to add the file to the `.gitignore`. That is why you will not find this file in the repo.

Make sure to adapt the `.env` file with your info, if you are using a database service.

Back to our *src/data/mongo_connection.rs*:

```rust
    let manager = MongodbConnectionManager::new(
        ConnectionOptions::builder()
            .with_host(&mongodb_address, mongodb_port.parse::<u16>().unwrap())
            .with_db(&database)
            //.with_auth(mongodb_user, mongodb_password)
            .build(),
    );
    match Pool::builder().max_size(64).build(manager) {
        Ok(pool) => pool,
        Err(e) => panic!("Error: failed to create database pool {}", e),
    }
}
```

At the end all this is all boilerplate. Notice the authorization part commented out here as well.

Let's see how *src/lib.rs* has changed since last time:

```rust
#[macro_use] use rocket::*;
use rocket_contrib::serve::StaticFiles;
use rocket_contrib::helmet::SpaceHelmet;
pub mod routes;
pub mod data;

pub fn rocket_builder() -> rocket::Rocket {
    rocket::ignite().attach(SpaceHelmet::default())
    .mount("/", routes![routes::ping::ping_fn])
    .mount("/api", routes![
        routes::user::user_list_rt,
        routes::user::new_user_rt,
        routes::user::info_user_rt,
        routes::user::update_user_rt,
        routes::user::delete_user_rt,
        routes::user::patch_user_rt,
        routes::user::id_user_rt
    ])
    .mount("/files", StaticFiles::from("static/"))
    .manage(data::mongo_connection::init_pool())
}
```

This is the whole file! As you can see, we do not need anymore the `Users` struct, as in order to save our users we are employing not a shared state, but a proper DB.

Therefore, we took out `.manage(Users::new())` and in its stead we are managing a `Pool` object, by init'ing it with `data::mongo_connection::init_pool()`.

And really, that is all we need.

## Connect the routes to MongoDB

Now let's work on our routes inside *src/routes/user.rs*.

Here we could go about with me explaining every little choice in the code, and why... But I feel this will not be profitable for your learning. You have the whole code in the repository, so I feel confident you could just copy&paste it!

However, I think that you are reading this tutorial in order to learn something useful to apply to your projects, not in order to learn to copy the code. Isn't it?

So let's start to see our `use` section:

```rust
use rocket::*;
use rocket_contrib::json::{Json, JsonValue};
use rocket_contrib::json;
use rocket_contrib::uuid::Uuid;
use rocket::response;
use rocket::http::{ContentType, Status};
use rocket::response::{Responder, Response};

use r2d2_mongodb::mongodb::bson as bson;
use r2d2_mongodb::mongodb as mongodb;

use bson::{doc, Bson};
use mongodb::db::ThreadedDatabase;
use mongodb::coll::options::{ReturnDocument, FindOneAndUpdateOptions};

use crate::data::db::{User, InsertableUser, ResponseUser, UserPassword};
use crate::data::mongo_connection::Conn;
```

We do not need anymore `use crate::Users`, neither the `rocket::State`. In fact our `Conn` already implements a `FromRequest`.

What we need in addition to what was already there, is:

- `doc!` macro and `Bson` type, from `bson`; however, we are not using the bson functions from the [bson crate](https://crates.io/crates/bson), but those exposed by `r2d2_mongodb::mongodb::bson`, because the `r2d2_mongodb` version is not up to date with the `bson` crate, and can give problems. Same thing as `mongo`, we are using the crate version exposed by `r2d2_mongodb`.

- As we already said, we are using the `mongodb` crate exported by `r2d2_mongodb` as well. Thus, we alias this to `mongodb`, and we get the `ThreadedDatabase`, which is used by `r2d2`'s pool; we import also some Options types for use in calling the MongoDB main functions.

- In addition, we'll use the `crate::data::mongo_connection::Conn` recently created.

Next we define a constant for the collection name:

```rust
const COLLECTION: &str = "users";
```

We should get it in each function with the `env::var()`, and that is the professional way to go about it, but for now let this suffice (feel free to modify your code though, which is a useful exercise).

We impl an additional error for `ApiResponse`, because we will use it a lot.

```rust
pub fn internal_err() -> Self {
    ApiResponse {
        status: Status::InternalServerError,
        message: json!("Internal server error"),
    }
}
```

Let's face it: if the connection between our server and MongoDB fails, we'd better not let this out to the world, but send a "stupid" `500` `"Internal server error"`. Better this than explain, and in doing so, expose info that could potentially lead to security breaches.

Let's start with the first route:

```rust
#[get("/users")]
pub fn user_list_rt(connection: Conn) -> ApiResponse {
    let user_coll = &connection.collection(COLLECTION);
    match  user_coll.count(None, None) {
        Ok(res) => ApiResponse::ok(json!([res])),
        Err(_) => ApiResponse::internal_err(),
    }   
}
```

We get `connection: Conn` as parameter. Then we select the collection with `&connection.collection(COLLECTION)` and assign it to `user_coll`.

Then we need to count how many documents (user records) are there in the collection with `.count()`. The function takes two `Options` to which we pass `None` to stick with the defaults.

We `match` over the `Result` as we said earlier we would often do, and we either return the count inside a vector, or an "Internal server error", with `ApiResponse::internal_err()`. The reason is that the `.count()` method fails mostly because of connection errors, so we cannot even test over it.. However, we are not off the hook because of connection errors: we need to make sure that when it will fail (and someday it will, if the service gets used a lot), we'll fail gracefully. That is why at each possible unforeseen fail you'll see a `ApiResponse::internal_err()`.

`count()` is one of the many calls we can do over a collection. Check out [https://docs.rs/mongodb/0.3.12/mongodb/coll/struct](https://docs.rs/mongodb/0.3.12/mongodb/coll/struct.Collection.html): there is a complete list.

Of all these calls, for our routes we need: `insert_one()` to insert a user into the collection, `find_one()` to retrieve a user, either by its `id` or one of the field's value, `find_one_and_delete()`, to delete a user.

We'll need also `find_one_and_replace()` to update info on the user. In fact it's easier to replace a user with an updated version of it, than update it, with ` find_one_and_update()`. The problem is that in order to update info, we use custom functions, such as hashing the password, and  update the timestamp.

By the way, we need to modify `fn update_password()` and `fn update_user()` a little, in our *src/data/db.rs*. In fact, we will not simply update the original User, but we need to return the updated User:

```rust
pub fn update_password(&mut self, password: &String) -> Self {
    self.hashed_password = hash_password(password, &self.salt);
    self.updated = Utc::now();
    self.to_owned()
}
pub fn update_user(&mut self, name: &String, email: &String) -> Self {
    self.name = name.to_string();
    self.email = email.to_string();
    self.updated = Utc::now();
    self.to_owned()
}
```

Now it updates itself, and it returns the updated version, so we can use it to **replace** the older one inside the MongoDB.

Now for example, inside `update_user_rt()`, that updates `name` and `email`, first we search for the user with `user_coll.find_one()` then, after much `match`-ing, we parse the user inside the collection back to our struct `User` with `bson::from_bson(Bson::Document(found_user))` (which needs itself other matches).

We check also that the password to authorize the change is correct and at the end of this process we update the `User` with the `found_user.update_user(&user.name, &user.email)`. Then we can replace the user in the MongoDB with this one, using `find_one_and_replace()`

Things to notice:

- `find_one()` returns a `Result<Option<Document>>`; the Result is needed to check that the connection didn't fail (which we match with an `Err(_) => ApiResponse::internal_err()`). The `Result` wraps an `Option`, because in case the operation did not fail, it can happen that we are searching for a document that does not exist (think of wrong user id, for example); so if there's no such `Document`, the `find_one()` returns a `None`. If that happens, we match the `None` with `ApiResponse::err(json!(format!("id {} not found",  id)))`.
- Notice too, that `find_one()` accepts a BSON document (created with `doc!` macro) to specify what's been sought for. In this case the document is a JSON object that sets `_id` to the `id` of the user. Since we need the default options, we pass to it also a `None` as second argument.
- once we found the `Document` we need to parse it from `Document` to BSON, and from BSON to our `User`. In order to do so, we need to give hints to the compiler, especially that we need to parse a `User`; and we need to `match` it, because `bson::from_bson()` can possibly fail (if the document is malformed, again usually because of connection errors). So the hint we give to the compiler is `Result<User, _>`. We use `_` because we don't care here about the type of the `Result`.
- the inverse function, `bson::to_bson()` can also fail, so we need to `match` it as well (well, if this fails it's not a connection error, but probably a bug... but we match over it nevertheless).
- `find_one_and_replace()` can return either the original document, or the updated one. We need this last one, because we need to return the info, as it was this our API signature. So we construct first a `FindOneAndUpdateOptions`, and we set its `return_document` field to `ReturnDocument::After` (it must be wrapped in an Option).
- once we get back the updated User, we need to parse this again with `bson::from_bson`, and match everything. After this, we need to return not the User (with all its fields, such as `updated`, `salt`, and `hashed_password`), but a `ResponseUser` with our trusty `from_user()`.

All this is translated to the following code:

```rust
#[put("/users/<id>", format = "json", data = "<user>")]
pub fn update_user_rt(connection: Conn, user: Json<InsertableUser>, id: Uuid) -> ApiResponse {
    let user_coll = &connection.collection(COLLECTION);
    let id =  id.to_string();
    match user_coll.find_one(Some(doc! { "_id": id.clone() }), None) {
        Ok(find_one) => {
            match find_one {
                Some(found_user) => {
                    let found_user_doc: Result<User, _> = bson::from_bson(Bson::Document(found_user));
                    match found_user_doc {
                        Ok(mut found_user) => {
                            if found_user.match_password(&user.password) {
                                let insertable = found_user.update_user(&user.name, &user.email);
                                match bson::to_bson(&insertable) {
                                    Ok(serialized) => {
                                        let document = serialized.as_document().unwrap();
                                        let mut opt = FindOneAndUpdateOptions::new();
                                        opt.return_document = Some(ReturnDocument::After);
                                        match user_coll.find_one_and_replace(
                                            doc! { "_id": id.clone() },
                                            document.to_owned(),
                                            Some(opt)
                                        ) {
                                            Ok(updated_one) => {
                                                match updated_one {
                                                    Some(updated_user) => {
                                                        let updated_user_doc: Result<User, _> = bson::from_bson(Bson::Document(updated_user));
                                                        match updated_user_doc {
                                                            Ok(updated) => ApiResponse::ok(json!(ResponseUser::from_user(&updated))),
                                                            Err(_) => ApiResponse::internal_err(),
                                                        }                                                        
                                                    },
                                                    None => ApiResponse::err(json!(format!("id {} not found",  id))),
                                                }
                                            },                    
                                            Err(_) => ApiResponse::internal_err(),
                                        }
                                    },
                                    Err(_) => ApiResponse::internal_err(),
                                }
                            }
                            else { ApiResponse::err(json!("user not authenticated")) }
                        },
                        Err(_) => ApiResponse::internal_err(),
                    }
                },
                None => ApiResponse::err(json!(format!("id {} not found",  id))),
            }            
        },
        Err(_) => ApiResponse::internal_err(),
    }
}
```

As you can see, lot of code, with 7 `match`, and one `if` to take care of.
See if, by checking the code with the explanation above you can understand what's being done.

## The rest of the routes

If you understand the working of the code above, you'll get as a breeze both `info_user_rt`, which gets info on the user with the `id`, or `id_user_rt`, which gets the same info but searching by `email` instead.
Both of them use `find_one()`, the first with `doc! { "_id": id.clone() }`, and the second with `doc! { "email": email.clone() }`.

As for the other routes, these are the major things to notice:

- `new_user_rt()` first gets a `User` from the `InsertableUser`, then parses it to BSON. Once serialized to BSON, it transforms it to a MongoDB `Document`, and passes it to the `insert_one()` collection's method. This method *does not* return the user just inserted, but a `Result` - wrapped record containing the inserted document's `id` as `inserted_id` (or a `None` if it was not inserted). At this point, we use the `inserted_id` to search back the user with `find_one()`, and we follow from now on, the same pattern we have used before with the other routes that get info on the user (check the `update_user_rt` above).

- with `delete_user_rt()` we need first to `find_one()`, and then match the password (as with `update_user_rt`). When we have cleared that the password is correct, we use `find_one_and_delete()` which works just as `find_one()`, and returns also the recently deleted record At this point we end up following the same pattern as with `find_one()`.

- as for `patch_user_rt()`, with which we update the password, we match the presence of the new password (otherwise it's useless, and we can abort); then we `find_one()` and match the original password. Once we are authorized (with `User.match_password()`), we get an updated user, and we reinsert it with `find_one_and_replace()` as we did to update the user info. We do not need to return the info on the updated user in this case, but just the message `"Password updated"`.

And this is really all there is.

We can now run `cargo test` and proudly see all tests passing (if everything went smooth), even the new one in the *tests/persistency_test.rs*.

## Conclusions

It has been a lot of code, I hope you can navigate through the ocean of `match` to get to what is the core of the functions.

Next time we will improve something more, and upgrade our security a notch.

Stay tuned!