Highly available cross region application architecture leveraging MongoDB

Part 1

What am I trying to solve ?

Traditionally when we look at highly available applications that are spread across geographically distinct data centers, the database layer of such applications is more likely to be one that supports a master/master mode with low latency replication between them. The high level idea is that the application can write to a master that offers that lowest latency and the data will be available in other masters within a short span of time. The words low latency, short span of time etc. were chosen intentionally to ensure that I am not getting into the weeds like ACID vs non ACID etc .

We all would have seen a number of applications that is built on the above mentioned architecture. So you may ask , Why a specific discussion about MongoDB ?

MongoDB is a NoSQL based high performance database that supports only a single primary server (for a given dataset) at any point in time, well at least as of May 2021. A mongo cluster will usually have many servers in it, but only one master ,rest are all secondaries. While slaves can service read requests , all writes must be performed on the server that is the primary at that point.

This architecture of mongoDB presents a challenge when dealing with applications that are deployed as active in multiple data centers at the same time. The simplest choice would be to have the application instances running in various data centers to connect to the mongo primary , even if that means cross data center data traffic for N -1 data centers ,where N is the total number of data centers where the application is running.

What am I proposing ?

The option that I prototyped was to leverage MongoDB sharding and the improvements to sharding in MongoDB 4.2 and 4.4. The core idea revolves around sharding the required collections based on a key that represents the data center or region in which the application is running.

How does it work ?

The Crux of the idea is to use a non sharded collection that is very lean with only few attributes that servers two purposes

• To help maintain uniqueness of the primary application key (Without the need to enforce this uniqueness at the application layer)
• To locate the actual shard where a particular application data “record” is present so that the subsequent queries can use the region key. (In the traditional sharding deployment, the application really doesn’t need to care where the data resides as this is handled by the mongos and the config servers, but the application query has to always include the sharding key.)

Will a particular record be always in the same shard ?

Well , that depends . The idea is to have the data in the shard cluster whose primary is in the same data center as the application cluster processing a request. So if a particular data record is consistently processed by an application cluster in a data center that has to go across the network to another data center to get to the shard primary of that data record, it may be time to update the shard key value on that record and bring it to a local cluster. (Yes, ability to update the shard key value is the key feature in MongoDB 4.4+ that made this testing possible).

What is the ideal application type for this kind of sharding usage ?

Any application that would route traffic to a data center based on proximity or location bias would be an ideal candidate. For e.g I may be a ACME Bank user who is based out of Virginia , so most of the time I will be serviced by the ACME Banking servers location on east coast. Once in a while when I travel to west coast for fun or “work” like Gartner conferences or AWS re-Invent , my traffic might hit the ACME west coast servers, but the cross region traffic that the application has to deal with at that time is temporary. Now, let us say that I relocated to Hawthorne, Then ACME Servers would see that it is constantly having to do cross data center queries for me and could update my region key value to bring my user record local to the west coast shard.

In part 2 of this article , I will explain the test setup I used , some mongo commands that I used to set up sharding for my test and the nodeJS scripts that I used to test the functionality

I should mention that this is more of a thought experiment and I am by no means an expert at MongoDB.