Core FQL concepts part 2: Temporality in FaunaDB
Tags: serverless, fauna
Published: 2020-10-02 18:00:32

Author: Pier Bover
Publish Date: Sep 16, 2020

---

Today we're going to explore one of FaunaDB's most unique features: its temporal capabilities.

This series assumes you have a grasp on the basics. If you're new to FaunaDB and/or FQL here's [my introductory series on FQL](https://fauna.com/blog/getting-started-with-fql-faunadbs-native-query-language-part-1).

##In this article:
* Document history
* Querying document history
* Filtering history events
* Travelling back in time
* Recovering documents
* Careful with the spacetime continuum
* Use cases and considerations

##Document history
One of the main temporal features of FaunaDB is being able to record every change you make to a document. When this is configured, new document snapshots are created for every change and added to the document's history.

By default, when creating a collection we get 30 days of minimum document history:
```
CreateCollection({name: "MyCollection"})

{
ref: Collection("MyCollection"),
ts: 1598891726415000,
history_days: 30,
name: "MyCollection"
}
```
History will last for at least 30 days with the default **history_days** value, but there are no guarantees when the deletion occurs as FaunaDB periodically reclaims expired snapshots.

We can change the minimum default to any number of days that fits our use case:
```
CreateCollection({
  name: "CollectionWithAYearOfHistory",
  history_days: 365
})
```
If we wanted to retain all history until the end of times we'd just use a **null** value:
```
CreateCollection({
  name: "CollectionForever",
  history_days: null
})
```
It's also possible to expire all history by default by passing a 0 value:
```
CreateCollection({
  name: "CollectionNoHistory",
  history_days: 0
})
```
### Difference with ttl_days
When creating a new collection it's also possible to configure **ttl_days** (time-to-live days). This setting is not related to its history. Instead, it refers to how long you'd like to keep the current or active version of a document alive. This is most useful for ephemeral data such as sessions, caches, and so on.

Check the [documentation](https://docs.fauna.com/fauna/current/api/fql/functions/createcollection?lang=javascript) for more details.

###Security implications
It should be noted that having permissions to read a document doesn't automatically grant permission to read or modify its history. Your users will need dedicated permissions to be able to interact with the history of a document.

For more info on authorization in FaunaDB [check my previous article](https://fauna.com/blog/getting-started-with-fql-faunadbs-native-query-language-part-5).

##Querying document history
Before we go on, let's create some data:
```
> CreateCollection({name: "Lasers"})

> Create(
  Collection('Lasers'),
  {
    data:{
      color:"BLUE",
      crystal: "KYBER"
    }
  }
)
 
{
  ref: Ref(Collection("Lasers"), "275571113732342290"),
  ts: 1599063943620000,
  data: {
    color: "BLUE",
    crystal: "KYBER"
  }
}
```
Let's also modify the document we just created to get some history:
```
> Update(
  Ref(Collection("Lasers"), "275571113732342290"),
  {data:{color:"GREEN"}}
)

> Delete(
  Ref(Collection("Lasers"), "275571113732342290")
)
```
We can query a document's history by using the [Events()](https://docs.fauna.com/fauna/current/api/fql/functions/events?lang=javascript) function. Here's an example of what this history looks like:
```
> Paginate(
  Events(Ref(Collection("Lasers"), "275571113732342290"))
)
 
{
  data: [
    {
      ts: 1599063943620000,
      action: "create",
      document: Ref(Collection("Lasers"), "275571113732342290"),
      data: {
        color: "BLUE",
        crystal: "KYBER"
      }
    },
    {
      ts: 1599064033700000,
      action: "update",
      document: Ref(Collection("Lasers"), "275571113732342290"),
      data: {
        color: "GREEN"
      }
    },
    {
      ts: 1599064043550000,
      action: "delete",
      document: Ref(Collection("Lasers"), "275571113732342290"),
      data: null
    }
  ]
}
```
As you can see, **Events()** takes a document reference and returns a list of all the changes we've performed on that document. An important point is that this history is available even after the document has been deleted.

Instead of using **Events()**, we could also use the options of **Paginate()** to retrieve the history of multiple documents at once:
```
> Paginate(
  Documents(Collection("Lasers")),
  {events: true}
)

{
  data: [
    {
      ts: 1599063943620000,
      action: "add",
      document: Ref(Collection("Lasers"), "275571113732342290")
    },
    {
      ts: 1599064043550000,
      action: "remove",
      document: Ref(Collection("Lasers"), "275571113732342290")
    },
    // etc...
  ]
}
```
In this example, we are getting all the history of all the documents in the **Lasers** collection by combining [Documents()](https://docs.fauna.com/fauna/current/api/fql/functions/documents?lang=javascript) with **{events: true}**.

This approach is extremely powerful as you could, for instance, combine multiple indexes to fetch the history of documents based on complex filters, or even from documents from multiple collections. Definitely check out the [Paginate()](https://docs.fauna.com/fauna/current/api/fql/functions/paginate?lang=javascript) documentation for more info on this.

##Filtering history events
When querying the history of a document by using **Events()**, we get an array which can be manipulated just like any other array in FaunaDB.

For example here's how we could just get the **update** events:
```
> Filter(
  Select(
    ["data"],
    Paginate(Events(Ref(Collection("Lasers"), "275571113732342290")))
  ),
  Lambda(
    "event",
    Equals(
      Select(["action"], Var("event")),
      "update"
    )
  )
)
 
[
  {
    ts: 1599064033700000,
    action: "update",
    document: Ref(Collection("Lasers"), "275571113732342290"),
    data: {
      color: "GREEN"
    }
  }
]
```
We could also filter events by date:
```
> Filter(
  Select(
    ["data"],
    Paginate(Events(Ref(Collection("Lasers"), "275571113732342290")))
  ),
  Lambda(
    "event",
    Equals(
      ToDate(
        Epoch(
          Select(["ts"], Var("event")),
          "microseconds"
        )
      ),
      Date('2020-09-02')
    )
  )
)
 
[
  {
    ts: 1599063943620000,
    action: "create",
    document: Ref(Collection("Lasers"), "275571113732342290"),
    data: {
      color: "BLUE",
      crystal: "KYBER"
    }
  },
  {
    ts: 1599064033700000,
    action: "update",
    document: Ref(Collection("Lasers"), "275571113732342290"),
    data: {
      color: "GREEN"
    }
  },
  {
    ts: 1599064043550000,
    action: "delete",
    document: Ref(Collection("Lasers"), "275571113732342290"),
    data: null
  }
]
```
**Quick note:** We explored working with dates and times [in the previous article](https://fauna.com/blog/core-fql-concepts-part-1-working-with-dates-and-times). As a quick reminder, we're converting an integer to a **Timestamp** using [Epoch()](https://docs.fauna.com/fauna/current/api/fql/functions/epoch?lang=javascript), and then that timestamp to a **Date**, to be able to compare it with **Date('2020-09-02')**.

We could also use the **Paginate()** options to get the events after or before a certain date:
```
> Paginate(
  Documents(Collection("Lasers")),
  {
    events: true,
    before: Date("2020-09-03")
  }
)

> Paginate(
  Documents(Collection("Lasers")),
  {
    events: true,
    after: Date("2020-09-03")
  }
)
```
##Travelling back in time
The document history tells us that something happened at some point in time. What if we wanted to know the complete state of a document in the past? For example, after an update event or even in between history events?

We can use the [At()](https://docs.fauna.com/fauna/current/api/fql/functions/at?lang=javascript) function to do exactly that:
```
> At(
  1599064033700000,
  Get(Ref(Collection("Lasers"), "275571113732342290"))
)
 
 
{
  ref: Ref(Collection("Lasers"), "275571113732342290"),
  ts: 1599064033700000,
  data: {
    color: "GREEN",
    crystal: "KYBER"
  }
}
```
Much like a time machine, **At()** allows us to execute FQL queries in the past. Here we're using the **1599064033700000** timestamp from the **update** event to get the full document when we updated the color to **GREEN**.

**At()** sets the point in time for all reads in FQL for that query. We can use indexes, filter data, etc. It's exactly like querying FaunaDB at that time.

It doesn't work exactly like a time machine though. We can't really write changes in the past with it. If we try to use **At()** to make any changes at any other time than **Now()** we will get an error.

Let's create a new document to try this out:
```
> Create(
  Collection('Lasers'),
  {data:{color:"RED"}}
)

{
  ref: Ref(Collection("Lasers"), "275572558214988288"),
  ts: 1599065321160000,
  data: {
    color: "RED"
  }
}
```
And now let's add a couple of milliseconds to the creation timestamp and try to update the document in the past:
```
> At(
  1599065321170000,
  Update(
    Ref(Collection("Lasers"), "275572558214988288"),
    {data: {color: "PURPLE"}}
  )
)

Error: [
  {
    "position": [
      "expr"
    ],
    "code": "invalid write time",
    "description": "Cannot write outside of snapshot time."
  }
]
```
##Altering the spacetime continuum
While we cannot use **At()** to alter the past, it is possible to use [Insert()](https://docs.fauna.com/fauna/current/api/fql/functions/insert?lang=javascript) and [Remove()](https://docs.fauna.com/fauna/current/api/fql/functions/remove?lang=javascript) to modify the events in a document's history.

To demonstrate this, let's create a new document first:
```
> Create(
  Collection('Lasers'),
  {
    data:{
      color:"ORANGE",
      crystal: "QUARTZIUM"
    }
  }
)
 
{
  ref: Ref(Collection("Lasers"), "275678520646042112"),
  ts: 1599166374745000,
  data: {
    color: "ORANGE",
    crystal: "QUARTZIUM"
  }
}
```
Now let's see what happens if we insert a create event 1 second before we create that document:
```
> Insert(
  Ref(Collection("Lasers"), "275678520646042112"),
  TimeSubtract(Epoch(1599166374745000, "microseconds"), 1, "second"),
  "create",
  {
    data: {
      color: "YELLOW",
      crystal: "QUARTZIUM"
    }
  }
)
 
{
  ts: 1599166373745000,
  action: "create",
  document: Ref(Collection("Lasers"), "275678520646042112"),
  data: {
    color: "YELLOW",
    crystal: "QUARTZIUM"
  }
}
```
Keep in mind that the history is related to a document reference. Be careful not to use the wrong reference with **Insert()** and **Remove()**, otherwise these changes will have unintended consequences.

Let's see how the insertion of the **create** event affected the history of the document:
```
> Paginate(Events(Ref(Collection("Lasers"), "275678520646042112")))
 
{
  data: [
    {
      ts: 1599166373745000,
      action: "create",
      document: Ref(Collection("Lasers"), "275678520646042112"),
      data: {
        color: "YELLOW",
        crystal: "QUARTZIUM"
      }
    },
    {
      ts: 1599166374745000,
      action: "update",
      document: Ref(Collection("Lasers"), "275678520646042112"),
      data: {
        color: "ORANGE"
      }
    }
  ]
}
```
The new **create** event we just inserted is where we expect it to be, one second before the initial document creation, but look what happened with the last event in the array. It used to be a **create** event but it has been now converted to an **update** event.

In any case, if we now get the latest version of the document we can see it is unchanged:
```
> Get(Ref(Collection("Lasers"), "275678520646042112"))
 
{
  ref: Ref(Collection("Lasers"), "275678520646042112"),
  ts: 1599166374745000,
  data: {
    color: "ORANGE",
    crystal: "QUARTZIUM"
  }
}
```
##Recovering deleted documents
I'm sure I'm not the only one that has wanted to go back in time to restore something that was deleted by accident. FaunaDB offers a couple of ways to do this.

###Removing the delete event
We've seen that deleting a document creates a **delete** event. Could we use **Remove()** to undo that?

First let's delete our orange laser:
```
> Delete(Ref(Collection("Lasers"), "275678520646042112"))
```
Now let's get all the delete events for this document:
```
> Filter(
  Select(
    ["data"],
    Paginate(Events(Ref(Collection("Lasers"), "275678520646042112")))
  ),
  Lambda(
    "event",
    Equals(Select(["action"], Var("event")),"delete")
  )
)

[
  {
    ts: 1599177868380000,
    action: "delete",
    document: Ref(Collection("Lasers"), "275678520646042112"),
    data: null
  }
]
```
Cool, so let's use **Remove()** to remove the delete event:
```
> Remove(
  Ref(Collection("Lasers"), "275678520646042112"),
  1599178388870000,
  "delete"
)
```
We can confirm the removal of the **delete** event by checking its history:
```
> Paginate(Events(Ref(Collection("Lasers"), "275678520646042112")))
 
{
  data: [
    {
      ts: 1599166373745000,
      action: "create",
      document: Ref(Collection("Lasers"), "275678520646042112"),
      data: {
        color: "YELLOW",
        crystal: "QUARTZIUM"
      }
    },
    {
      ts: 1599166374745000,
      action: "update",
      document: Ref(Collection("Lasers"), "275678520646042112"),
      data: {
        color: "ORANGE"
      }
    }
  ]
}
```
We now can even **Get()** the document as usual:
```
Get(Ref(Collection("Lasers"), "275678520646042112"))

{
  ref: Ref(Collection("Lasers"), "275678520646042112"),
  ts: 1599166374745000,
  data: {
    color: "ORANGE",
    crystal: "QUARTZIUM"
  }
}
```
##Bringing data back from the past
Another option to recover data is basically reading it from the past using **At()** and inserting it again back into the present.

Let's create a new laser to start with a fresh document:
```
> Create(
  Collection('Lasers'),
  {
    data:{
      color:"BLUE"
    }
  }
)
 
{
  ref: Ref(Collection("Lasers"), "275752290343715347"),
  ts: 1599236727185000,
  data: {
    color: "BLUE"
  }
}
```
Let's delete our document and check its history:
```
> Delete(Ref(Collection("Lasers"), "275752290343715347"))

> Paginate(Events(Ref(Collection("Lasers"), "275752290343715347")))
 
{
  data: [
    {
      ts: 1599236727185000,
      action: "create",
      document: Ref(Collection("Lasers"), "275752290343715347"),
      data: {
        color: "BLUE"
      }
    },
    {
      ts: 1599236793325000,
      action: "delete",
      document: Ref(Collection("Lasers"), "275752290343715347"),
      data: null
    }
  ]
}
```
We could now use the timestamp of the **create** event with **At()** to read it:
```
> At(
  1599236727185000,
  Get(Ref(Collection("Lasers"), "275752290343715347"))
)
 
{
  ref: Ref(Collection("Lasers"), "275752290343715347"),
  ts: 1599236727185000,
  data: {
    color: "BLUE"
  }
}
```
And finally restore the data before deletion, by simply creating a document with the same reference and data:
```
> Let(
  {
    deletedDoc: At(
      1599236727185000,
      Get(Ref(Collection("Lasers"), "275752290343715347"))
    ),
    ref: Select(["ref"], Var("deletedDoc")),
    data: Select(["data"], Var("deletedDoc"))
  },
  Create(
    Var("ref"),
    { data: Var("data") }
  )
)
 
{
  ref: Ref(Collection("Lasers"), "275752290343715347"),
  ts: 1599237181566000,
  data: {
    color: "BLUE"
  }
}
```
**Quick note:** [Let()](https://docs.fauna.com/fauna/current/api/fql/functions/let?lang=javascript) not only allows us to format output objects but also execute any output query like we're doing here after having collected the necessary data.

Let's check what happened to the document's history:
```
Paginate(Events(Ref(Collection("Lasers"), "275752290343715347")))
 
{
  data: [
    {
      ts: 1599236727185000,
      action: "create",
      document: Ref(Collection("Lasers"), "275752290343715347"),
      data: {
        color: "BLUE"
      }
    },
    {
      ts: 1599236793325000,
      action: "delete",
      document: Ref(Collection("Lasers"), "275752290343715347"),
      data: null
    },
    {
      ts: 1599237181566000,
      action: "create",
      document: Ref(Collection("Lasers"), "275752290343715347"),
      data: {
        color: "BLUE"
      }
    }
  ]
}
```
As you can see, even though we created a new document, the new **create** event has been appended to the previous history for that reference. This is happening because the history is actually associated with a document reference.

##Use cases and considerations
Now that we've seen an overview of the temporal features in FaunaDB let's examine a couple of common use cases in more detail.

###Safe delete
It's very common in databases without history or versioning to follow a couple of patterns to prevent data loss. One of those patterns marks data as deleted but keeps it around in its original place. This practice is usually called "soft delete". Another alternative is to have a "recycle bin", that's a table or a collection where you move the data to be deleted instead of actually deleting it.

Both of these patterns have pros and cons, and you can certainly implement them with FaunaDB. Although, unless you need specific features, the simplest option is certainly to delete documents as usual and use the temporal features we just saw to keep deleted data around for as long as you need it.

There are two major points to consider though:

1. Fauna stores every single document change in its history, which occupies storage space. It might be overkill for simple safe delete at scale if you don't need document versioning.
2. Since querying the history returns an array of events, you won't have as much flexibility as you would have by storing the deleted documents in a dedicated collection. For example, by using a "recycle bin" collection approach, you could use indexes and filter deleted documents by user, deletion reason, etc.

###Version control
Another major use case for the temporal features of FaunaDB is storing and keeping track of content changes. Let's say you're working on SpaceDocs, the next app for collaborative writing of spacey documents. You obviously need a version history feature to keep track of changes.

Storage shouldn't be a concern here, since you would be storing all of the versions anyway, say in a **DocVersions** collection. But, again, if you used the history to keep older versions around you wouldn't be able to use common FQL features such as indexes. Maybe this would limit the kind of features you can build in the future.

Again, it all depends on your use case.

##Conclusion
So that's it for today. Hopefully you learned something valuable!

In the following article of the series, we will continue our space adventure by checking on aggregation queries and other features.


If you have any questions don't hesitate to hit me up on Twitter: [@pierb](https://twitter.com/PierB)