I think it started somewhere in the 80s as a pragmatic solution: ‘to prevent errors in production, development teams started working with copies of production databases in separate environments’. Why not?
- Environments were often small and manageable and could therefore be simulated well;
- Production data contained the most relevant data for testing systems;
- The risks were considerably smaller and there was little to no legislation.
I imagine that during that period, a bug or error in the development of the system caused an error to cause a production system to crash completely. The logical response? This must be prevented in the future! From now on, copies of production databases were made to use in a separate environment for further development and testing of the system. This is how the first teams were created that started working according to the DTAP method, a method that helped them to limit risks in software releases.
In my previous article I touched upon why I find test data strategically important. In this article I will delve deeper into what good test data means and why it is not production data. I will do this in the following paragraphs:
Why a copy of your production data is not a good idea anymore
Somewhere around the turn of the millennium, this approach slowly became untenable. Agile and DevOps accelerated the development cycle and made testing a continuous activity, while at the same time the amount of systems, data and technologies being worked with increased. The stakes have also increased and with them the privacy and security requirements have increased.
And yet teams want to continue testing with realistic data. And that’s where the problem lies: using a copy of production is often not safe or legally permitted, but it is the most realistic data you have.
Smarter testing without using production data
The key lies in better test data management. With techniques such as data anonymization and synthetic data generation, we can use realistic test data without infringing on privacy.
Data anonymization processes real data in such a way that it can no longer be traced back to individuals, but retains the correct structure and variation (internal link).
Synthetic data generation, on the other hand, creates completely new data that statistically resembles the real data, without ever using personal information.
This way, the test environment remains representative and we meet the strictest compliance requirements.
Complexity and risks are increasing; now is a good time for test data management
Because we increasingly use software for more intimate purposes such as: healthcare matters that we arrange online, internet banking and civil affairs, developing good software in a safe and responsible manner is becoming increasingly important. In addition, the interests and associated risks are becoming increasingly greater. All this happens while the work and its complexity continue to increase. With the ‘recent’ arrival and growth of cloud, SaaS and AI, the complexity of test data is growing even more.
Anyone who does not start now with getting a better grip on test data management will soon be increasingly stuck…. Not only technically, but also strategically. Because organizations that can smoothly adopt new technology are more agile, competitive and safer.
The way we handle test data must evolve with the new way of working, technologies and the stricter safety requirements and expectations from our governments, customers and consumers.
We should not be copying production databases anymore instead we should all strive to use proven test data management methods like data anonymization and synthetic data generation, these are not a luxury but a necessity: it is currently the best way to permanently balance speed, quality and compliance.