Building a horizontally scalable, fault tolerant stock application – Part 2

Let’s think about how we want to interact with our stock system. I think it needs to perform these basic functions first of all:

  • Register/book in stock, be it a new product or one already held
  • Track the current number of units held for a particular product
  • Reserve units for sale
  • Sell/remove units

Now these functions match pretty well with a traditional CRUD application, meaning REST principles should make this API quite simple. There are caveats we’ll need to consider later on such as obviously we can’t reserve or sell units we don’t have. For now i’m just going to focus on the API.

Before reaching a decision on what the API looks like, let’s consider how we’re going to query it. SKU is a fairly ubiquitous term and products have long had unique identifiers on to help us with this. So we’ll expect that if stock is being registered or booked in, a SKU will be provider to us and we won’t need to create our own unique references to help here.

With that in mind, we could start off with:

HTTP Method Endpoint Purpose
GET /sku/{sku} Query a SKU
POST /sku/{sku} Book in units against a SKU
POST /sku/{sku}/reserve Reserve units, this will return a reference for our reservation
GET /sku/{sku}/reserve/{reservation} Query a singular reservation
POST /sku/{sku}/reserve/{reservation} Complete a reservation

If we flesh this out initially using akka http, we can compose a small api using a nested tree structure.

Lets use the akka http api to construct an initial representation of the API, I’ll just update the akka demo code for now so we can demonstrate it working at the console.

Running this using `sbt run` should mean the we can replicate this behaviour

If we leave our API there for a moment, let’s consider the events that are going to define our system behaviour. Going from the list at the top, the things that can change state are:

  • Booking In
  • Reserving
  • Completing reserverations

Modelling that in scala I came up with the below. I’ve created command and event pairs. We’ll be sending the persistent actor the command which is the imperatively named classes. Should we decide to accept them, we’ll persist the past tense named event and update the state after acceptable levels of persistence.

On the next part, i’ll create the actor that models validation and persistence of these events

Building a horizontally scalable, fault tolerant stock application – Part 1

Let’s build a stock application.

The goal of this will be to keep complete auditability of the stock while keeping the application scalable. Many enterprise systems for stock keeping use the same model, have the same problems and reach the same bottlenecks. Just because everyone does it, doesn’t mean it’s right.

A typical stock system is built backed onto an ACID compliant vertically scalable database. They use read and write locks liberally initially for querying, reservation and allocation. As demand on the system grows, the lock contention increases and performance improvements are sought. The hardware is upped; dirty reads are introduced; designed are changed to perform fewer and smaller queries. Application errors increase, the numbers becomes inaccurate. It all silently spirals out of control.

There are better ways of solving this problem while gaining a host of other benefits.

What we’re going to build is an event sourced, eventually consistent application. It will know the complete history of your stock – not just the current stock level. It will be able to report stock levels even if the database dies. It will horizontally scale and be capable of recovering in the event of hardware failures.I’m going to presume you have the following tools setup:

So let’s start, create a new directory to work in and create our build file and supporting folder structure.

When we run SBT from our root directory, it’s going to look for certain files by default. One of these is build.sbt and another is project/

project/ helps SBT understand what version of SBT you want to run

build.sbt starts our project definition

We’re going to need some help to make all this work so we’ll add akka-persistence for our event sourcing framework. There’s good patterns for implementing event sourcing yourself in plenty of DDD books but akka persistence shines at abstracting a lot away for you. I’ll also add akka-http as we’ll need some way of triggering commands and interacting with our application. I’m using Martin Krasser’s Cassandra plugin here instead of akka-persistence directly as that will be the persistence plugin i’ll be using.

And then finally to show our build is at least working i’ll take the simple example app from the akka documentation and bung it in Main.scala. I have changed it a little as the example was using the scala-xml contrib library and seemingly missing an import.

So hopefully, from sbt you can do the command run and you’ll get a localhost:8080 application that responds to the resource /hello with a plain text response.

Next time we’ll start the API and the corresponding Actor events to go with it.

Testing Akka Persistent Views

I’ve been thinking about testing Akka Persistent Views recently.

PersistentView: A view is a persistent, stateful actor that receives journaled messages that have been written by another persistent actor. A view itself does not journal new messages, instead, it updates internal state only from a persistent actor’s replicated message stream.

The problem for me is coupling. For a PersistentView to be of any use, it needs to be coupled with a PersistentActor. It needs to match the right events and have the correct persistentId. Then of course the behavior needs to be correct.

To assert these conditions, your test needs to verify the pair together. So previously i’ve tended to treat my intended unit testing of a view as more of an integration test. Testing both sides together as real actors and then allowing for sleeping in the tests.

Now, I believe in good unit tests. They should be quick to run, quick to write, small and readable. I didn’t mind extending my tests to include both the persistent actor and view together because:

  • It proved the functionality worked. The unit is broken if the coupling is broken.
  • The tests were still quite small, lightweight and quick to run
  • Akka is still evolving and persistence is still marked experimental

This method though creates an entirely worse problem. Test reliability. If you’re testing both a PersistentActor and a PersistentView together, there’s no guarantee on when the update will occur. How quickly the view actually performs the update is non-deterministic and can rely on the right conditions that your test can’t control.

TestKit provides a variety of functions for waiting for messages, but unless your view emits something, you don’t really know if a view has been updated or not when you query it. Do you wait long enough that you think the view MUST have updated by now. Get it wrong and your build fails. Flakey tests aren’t worth the effort, nobody trusts them anyway.

So what’s the solution?

For now I think the best way forward is just unit testing the view and I suggest avoiding isPersistent calls to aid you in that. Create a PersistentView test, fire the events at the view, then test that like any other actor. The beauty of the model is that if that coupling is broken, your system hasn’t lost any information – just your ability to display that information.

I don’t think I’d even bother trying to creating integration tests that assert purely that coupling is correct. I’d allow a behavior / system test verify that things are wired together properly.