Stateful Testing Systems

Most test suites assume a fresh, disposable world every time they run. Spin up a database, seed some fixtures, run the assertions, throw it all away. That model works beautifully right up until the thing you need to test doesn’t fit inside a single process lifetime.

Take a system that sends a real verification email as part of signing up, and enforces a real rate limit on how many signups can happen per hour. There is no bypass for either of those, because bypassing them would mean not testing the real thing. You have to actually wait for the email to arrive, and you have to actually respect the rate limit. Neither of those fits into “run a function, assert on the result, tear everything down.”

Hexagonal Architecture Design

The hexagonal architecture helps to build robust and change friendly code. I have used multiple different architecture paradigms over the last 20+ years of writing software. For me the hexagonal approach is the best when it comes to modern software engineering. It can be used in all programming languages and helps to have a sustainable software development effort over a long time.

Summary

The hexagonal architecture has many representations. It is usually displayed as a hexagon that has three layers. Next to the layers there is an east-west/left-right split.

Reducing the database load

In the previous post we discussed the importance of the unique ID for every record. Still we will update the records multiple times a day even if we don‘t change anything. Remember we scrape the data. Assuming 1Million records with 10 different sources e.g. and a scrape interval of 5 minutes we easily have a database load of 1M * 10 sources every 5m which equals to 120M rec/h which equals 33K req/s which has to potential to overload the database depending on the technology.

Unique ID

Deduplication of the data acquired in the distributed data pipeline is accomplished by using a common id. All records that are related to the same physical location need to end up having the same unique id.

Starting out, there are no unique ids. We only have data sources that can’t contribute data to any existing record. Looking back to the source A example from the previous post, we will start with only:

Distributed Multi Source Continuous Data Pipeline

In this series I will detail a solution to a common problem with distributed data aggregation. We want to build a web application that displays current price and location data for EV charging stations on a map. The data is scraped from websites, sourced by the government or similar data sources. The here described solution has been in production for years. So it is known to solve the above problem.