Once we were confident that we had the data under control, it was time to start building out the new application. This project required that we use PostgreSQL to store the data and Perl to program the logic. Although Perl is sometimes criticized for being cryptic and difficult to maintain, we found it to be a good test of our belief in best practices. By applying well-established design patterns and test-driven development techniques, we produced an application that has proven to be rock solid.

While cloud-based infrastructure might seem like an obvious choice for delivering a high availability service, this project’s technical requirements didn’t initially allow for that.

Instead, we colocated servers with different service providers in multiple geographic regions for redundancy, we used Git to roll out code changes to all production servers at once and we used PostgreSQL’s own mirroring capabilities to achieve near real-time duplication of data writes across multiple servers at once.

Finally, for heightened security, we hardened all of the servers with Bastille Linux, and we secured the filesystems with Tripwire. We continually monitor server logs for anomalies, and we regularly perform penetration tests and vulnerability scans to keep holes plugged.