DBSubsetter

2017

DBSubsetter is a tool I wrote for taking a logically consistent subset of a relational database. It is written in Scala and relies heavily on the excellent Akka Streams library. It is free and open source software, with the source code and documentation available here.

To use DBSubsetter, you provide it with a SQL “where clause” which specifies a set of rows from one or more tables. It then introspects the schema of your database and follows foreign key constraints, recursively fetching the “parents” and, optionally, the “children” of those rows.

This is mainly useful for being able to give each member of a team of developers their own, smaller copy of a production database for them to develop against, although there are some other uses for it as well including data archival, etc.

Consistent subsetting is a problem that is trickier to solve than it seems at first, especially if you want the solution to be highly performant. The following are a couple of interesting points about the project.

In order to achieve a “recursive” algorithm without resorting to actual recursion, DBSubsetter contains a “feedback loop” in Akka Streams. The main challenges of feedback loops are preventing deadlocks due to backpressure and getting the stream to properly terminate when all outstanding work is done. Getting the program to complete was an interesting exercise that turned out to be possible using a combination of statefulMapConcat, takeWhile, and mapConcat graph stages. I wrote more about this here on the Akka User Mailing List.

It attempts to be vendor-agnostic and support MySQL, Postgres, and Microsoft SQL Server out of the box. This turns out to be possible but slightly difficult! One of the interesting parts of this project was using Docker to run the same tests against all three types of database, which often turned up issues present in just one of the three databases that would otherwise have gone unnoticed.

DBSubsetter

Advertisement

Mock Cryptocurrency Mixer

2017

I recently wrote a mock Cryptocurrency Mixer. The focus of this project was on the logic of how to mix the currency randomly (it was not concerned with the underlying cryptography).

The teck stack included:

This project is closed source for now (for fairness to other candidates, because it was written as part of a job interview), but I can share the source code on request.

crypto-currency-mixer

 

Materialized Views

2014

Extends ActiveRecord::Migration with methods for creating auto-updating materialized views in Postgres. Performs gold standard tests to check if a materialized view is up-to-date with its unmaterialized version.

This project is open source. View source code.

intouchsys

2014

A custom Rails app I built and sold to my old company, InTouch Manufacturing Services Ltd. It now underlies nearly every aspect of their operations including client-management, scheduling services, creating photo-rich PDF inspection reports, invoicing, and calculating commissions.

Expense Manager

2014

A Rails app for tracking company expenses. Interesting aspects include use of the facade pattern to organize data for a complex dashboard, scalable spreadsheet-like searching and sorting backed by Apache Solr, and date-based (rather than id-based) SQL joins.

This project is open source. View Source Code.

Screen Shot 2017-09-27 at 6.11.46 PM