Rewind or restore test data?

In the world of software testing, predictability is paramount. Testers and developers strive for precise, repeatable results, ensuring the reliability and quality of their applications. However, achieving this predictability can be a formidable challenge, particularly when it comes to managing and resetting test data. Imagine you’re in the midst of regression testing, and you need to ensure that the data you’re working with is consistent, accurate, and reflective of your pre-test state. You don’t want the actions of other testers or the complexities of your application’s data model to throw off your results.

This recurrent challenge has led to a quest for solutions that allow for the manipulation of specific test cases without affecting others or the need to reset the entire test dataset. In the past, we discussed two strategies: the ‘rewind’ and ‘restore’ methods. However, technology doesn’t stand still, and now, we have a game-changer in the realm of regression testing – database virtualization.

The rewind strategy

One of the ‘old fashioned’ strategies is to rewind the user actions performed during the test. Imagine a situation in which you successfully Added customer ‘John Broke’ as part of your latest interface test of the Customer Registration screen. After the test you functionally Delete ‘John Broke’ using built-in application logic.  However, are you aware of any possible side-effects of your application? Maybe your application has a business rule implemented that keeps track of customers who are ‘deleted’ without ever having ordered something and are therefore blacklisted in a dark corner of your database.  The next time you want to add ‘John Broke’ as a customer your application will deny you because of the applications’ business rule enforcing the blacklisting during your ‘believed to be safe and sufficient’ attempt to reset (‘rewind’) the data to its pre-test state.

The application is now reacting differently to the test compared to the first time and you are just about to find out that functionally deleting a customer does not bring your test data back to the state it was in before the respective customer was added. If the customer’s registration in the blacklist table were the only side-effect to compensate for in order to get the test data set back into the pre-test state, you could try to technically remove that record from the table(s) the application is using to manage the blacklist information. Chances are, however, that the business logic and the data model are complex black boxes, or that you as a tester may regard them as such.

It’s also true to say that any rewind logic which you may design must also be failsafe. There has to be a way to recover the recovery process in the event of failure, otherwise your database will be left in an inconsistent state where the only recovery option is to restore from backup or regenerate the subset.  This is a vicious circle where the rewind option can take more time in planning, coding and implementation than the original application feature that is being tested!

The restore strategy

Clearly, it is far safer and easier to use the restore test data strategy which means removing all the post-test content from the database and subsequently restoring the pre-test data set from a backup copy. It goes without saying that this strategy benefits from having smaller test sets. In fact you and your colleague testers/developers will benefit, as well as your organization which ultimately pays the cost of testing and test data provisioning. You should also consider the difference it will make in terms of accumulated response delay times and the effects on data storage and data processing when compared with testing against full-size copies of the production environment.

Database Virtualization

The Game Changer

At its core, database virtualization is a cutting-edge approach that allows testers and developers to have a deeper level of control over their databases. It involves creating virtualized copies of databases, also known as virtual copies, which can be manipulated and managed independently of the primary database. This innovation not only empowers testers but also provides a safeguard against unintended data modifications, ensuring the integrity of your test data throughout regression testing.

The Three Pillars of Database Virtualization

Let’s dive into the three key pillars of database virtualization:

1. Versioning: A History of Your Database

Versioning is at the heart of database virtualization. Every change made to the database is tracked, recorded, and preserved as a separate version. This means that for each modification, whether it’s data insertion, updates, or deletions, a new version is created. As a tester or developer, you can view and access the database at any point in time by simply selecting the version you want. This feature offers several benefits, including the ability to:

• Roll back to previous database states, ensuring that you can reset the data to a known pre-test state.
• Analyze the impact of specific changes by comparing different versions.
• Maintain a history of database modifications, aiding in debugging and audit trails.

2. Snapshotting: Freezing Time in Your Database

Snapshotting is a powerful aspect of database virtualization. Think of snapshots as frozen moments in time for your database. Whenever you need to create a checkpoint, you can take a snapshot of the current database state. These snapshots are read-only copies, preserving the data exactly as it was at the moment the snapshot was created. This feature offers the ability to:

• Create stable, consistent test environments by using snapshots as starting points for your testing activities.
• Ensure that your test data remains unchanged and predictable throughout regression testing.
• Quickly and efficiently reset your database to a known state, eliminating the need for time-consuming data reconfiguration.

3. Restoring: Bringing Your Database Back in Time

Restoring is the final piece of the puzzle. When issues arise or data inconsistencies are discovered during testing, database virtualization simplifies the process of restoration. If you need to roll back your database to a specific version or snapshot, the restoration process is swift and accurate. This feature provides the capability to:

• Revert the database to a known, consistent state, ensuring your test data reflects a pre-test condition.
• Correct any unintended data changes with ease, minimizing downtime and disruption to your testing activities.
• Maintain the integrity of your test data and improve predictability in regression testing.

Conclusion

Ultimately, the choice between data rewind, data restore, and database virtualization should be based on your specific testing needs, resource availability, and the complexity of your database. Striking a balance between simplicity and precision is the key to efficient regression testing. Consider the nuances and requirements of your testing environment, and be prepared to adapt your strategy to ensure that predictability and reliability remain at the forefront of your testing efforts. In the ever-evolving landscape of software development, the right choice can be the linchpin to success in your regression testing endeavors.