Apps that allow users to generate data while being offline (Trello, for example) generally provide a better user experience. They must resolve a challenge however: because the database schema naturally changes as the app evolves, the server must support incoming data generated on an old schema version. How? Here is one way of doing it.

The challenge

At Peek Vision, the company I work for, we collect eye health data from patients in rural areas with little or no connectivity. This means that devices must be able to collect data while being offline. And devices stay offline for surprisingly long periods of time: usually several days, often a couple weeks and sometimes even months.

When devices finally go online they send the data they have collected to the server. Which, like most web APIs, performs validations on it: if the incoming data doesn't have the expected format it will be rejected. Nothing unusual. But turns out that, as the app evolves and new software is released, the expected formats can change, potentially rendering invalid data collected on offline devices running on a previous software release.

Having hundreds of active users there will hardly be any moments where all devices have synced their data. Therefore we needed a mechanism to allow data generated on a previous release to be accepted by the server running on a newer release. We called that mechanism on-the-fly migrations, or OTF migrations for short.

The idea behind OTF migrations is quite straightforward: intercepting arriving data before validating it and applying required migrations if necessary. OTF migrations can be implemented at either global level or endpoint level. Personally, I find it cleaner to go for the endpoint level, but both approaches are equally valid:

• A global interceptor defines a single entry point for migrations, making endpoints migrations agnostic, but it requires checking the request method (we will only want to migrate request that have a body) and evaluating the requests' URL to decide which migrations need to run on a given payload.
• Endpoint-level interceptors split the migrations into several entry points, making endpoints migrations aware, but it produces simpler interceptors code.

Hands on

Consider a Trello-like app that allows creating notes in offline devices. The notes contains just a text field and the server has an endpoint that receives one note at a time, to which offline notes are posted when devices go online. This is what a NestJS implementation could look like, plus a sample client call:

The implementation will obviously vary depending on the server framework. Here I'm using NestJS but the core idea is applicable to any framework.

So far so good. Now it's time to introduce some breaking changes 💣 Let's assume that, at some point, we decide the notes will have a title and a description instead of a single text field.

We will change the database schema accordingly and write a script to migrate the existing data in the database. We will update the app as well to post the notes with a title and an optional description. Not a trivial change, but simple enough:

Note however that some devices might have generated offline data using the old schema (i.e. a note with a single text field), and they might not have synced the data by the time we deploy the changes to the live environment. In those cases the incoming notes will result in a validation error:

Here is where OTF migrations will come handy. In a nutshell, we want to manipulate the request body before the class-validator logic kicks in. There are several ways to achieve this in NestJS and here I'll use interceptors.

We could use middleware as well, but I find that slightly more obscure for this use case, as the Nest middleware must be applied to modules, whereas interceptors can be applied to specific endpoints.

An interceptor class must implement the NestInterceptor interface, and it can then be applied to any endpoint via the UseInterceptors decorator. Here is what our interceptor could look like and how to use it in the note creation endpoint:

Simple yet effective: the notes generated in the old schema version will now be automatically migrated and accepted by the class validators 💃

This implementation is not very scalable however. Chances are that we will need more OTF migrations as the app evolves, having to add more UseInterceptors decorators, in the correct order, to each endpoint that requires OTF migrations.

To prevent that we can group the different OTF migrations that must be applied to each model in a single interceptor. An elegant way of doing so is to split the interceptor from the migrations themselves, turning each migration into a simple function that will be invoked from the interceptor. Beautiful 💘

Another challenge of this implementation is that migrations will always run, even for data generated in the current schema version. This might backfire at us. Imagine we decide to add a boolean text property; the OTF migration will incorrectly replace the notes title. We need to prevent that from happening. We can determine which OTF migrations need to be applied to an incoming piece of data by:

• Including the current schema version to each entity generated on the app. The schema can be sent either as a separate parameter or as part of the entity itself. In this article, I'll use the latter.
• Specifying a target schema version for each migration.
• Comparing the schema version of the incoming data with the target schema of each migration.

Note that now we no longer need to check whether body.text exists; the schema condition ensures the incoming data will have a text property.

From now on the schema property included in the data will determine which OTF migrations are executed. Once the migrations have been applied, technically, we no longer need the schema property. We can either:

• Delete the property, so it doesn't end up in the database. Note we need to delete the property once all the potential OTF migrations have been applied.
• Keep the property, as it might be useful to migrate existing data in the database. In this case we need each migration to set the schema property on the entity to the migration's target schema.

Adding the aforementioned text property no longer messes with the incoming data:

Finally we can slightly improve the interceptors by automatically sorting the migrations based on the target schema and extracting the common behavior to a base class. This is not really necessary but, hey, we developers cannot help ourselves from refactoring, can we?

And that's it. A well tested OTF migrations architecture for NestJS/express servers. See this working demo repository for more details. Credit goes to Richard Evans, our lead developer, who will retire in September but whose contributions will remain in the company's legacy for years to come 🍻

Posts timeline