Simplifying test data and database changes
4 November, 2021 | Bert Nienhuis
At DATPROF, we love simplifying getting the right data, in the right place at the right time! The fun of working with lots of different customers, with various applications and database systems, is that you keep learning new challenges.
When talking to lots of organizations that want to implement some kind of test data management solution, they often ask; “How should we handle differences in data models in our development, test and production databases?”.
Managing such differences can be quite tricky. Often the development team is not in charge of rolling out database changes and database administrators must execute some scripts in a specific environment to get it all working.
Modern software principles like Agile, Scrum, DevOps are all pushing teams go to production faster than ever before with improved quality. Getting in control of your test data and your database changes is crucial to keep up and benefit from all the other efforts that you already have made.
A great product to manage database changes is Liquibase. This solution can automate the database change process: checking for differences between databases, updating database schemas for every environment, and even rollback changes. Liquibase offers a free open source community version, but there are also professionally supported editions that bring a lot of extra functionality to the table, such as quality checks for databases that ensure database developers can instantly validate their changes are safe and compliant before committing them.
When combining DATPROF with Liquibase, you can automate and simplify multiple steps in your test data provisioning pipeline. With the Liquibase diff command, you can easily compare the differences between your production and target database. Before creating a new subset from your production database, you can check if both databases use the same data model to efficiently pull the subset into the target database.
Combining the Liquibase snapshot command with the diff command allows you to capture the current state of your database so that you can compare it against a target database at a later point in time. This process can be automated within Runtime to, for example, notify the test data team that a template’s meta data is no longer an accurate reflection of the source or target database.
Upgrade your test database
After refreshing your test data, you can upgrade your test environment to a specific database model version, including creating extra tables, modifying columns or removing constraints.
Liquibase can automatically upgrade your database to a specific version of your data model. It checks the changelog of your data model version and determines which upgrade steps need to execute to upgrade the database model, automatically updating your test environment to the right version.
After you have executed your tests, you may want to refresh the data. Liquibase offers functionality to roll back all changes, or in the supported versions, cherry pick specific changes. This brings the database to a state where you can subset, mask or generate new data for your database and subsequently upgrade it again.