Some background…

Alessandro and I are working on a really cool and huge application (web application for the records) that is totally based on DDD, Event Sourcing and CQRS concepts.

I can safely say that we are really happy with what we have achieved so far.

But…

At the early beginning we decided to have 2 different ways to handle domain events, basically we differentiate domain events running in-process and domain events that should also run out-of-process. The end result was pretty interesting because using the Radical Message Broker we have internal asynchronous events flowing in-process and eventually we have the ability to “transform” an event to something that can also be broadcasted over the wire using NServiceBus.

Enter MongoDB

Within the ingredients of the recipe we are working on there is also MongoDB, the popular open source nosql database, that is really cool but has some drawbacks, especially for the .net developers, that sometimes leads you to tweets like the following:

image

to avoid working deeply with Map/Reduce I started to handle domain events, all sort of events, to de-normalize data on MongoDB so to achieve the same result that Map/Reduce leads to, but in a much much easier and sustainable way.

The problem

Runtime data de-normalization introduce the high risk of out-of-sync data: one of the de-normalizer fails and some data are not correctly de-normalized…

If this happens it is a much bigger pain that the one produced by a day-by-day working with MongoDB Map/Reduce.

The root cause

In our scenario the main bottleneck/point-of-failure is the in memory message broker because due to the asynchronous nature of the broker we lead to have lots of threads running, and if one of this thread fails we have no easy way, well no options at all, to recover from the failure.

Enter MessageQueue

So the question is: can we replace everything and only use NServiceBus?

why not :-) in the end NServiceBus does exactly the same thing, dispatches messages, but has all the advantages of a transactional and durable storage as the one provided out-of-the-box by the MessageQueue service.

So now we can have de-normalization processes that fails, and if they fails the message (event) that triggered the de-normalization is simply put in the error queue, we can solve the bug and reply the message.

We moved from a scenario with no recovery options to a scenario where the only problem is when the recovery will occur.

A new step toward the “forget-data-corruption” has been made, and it was really easy, really.

.m