Rewind or restore test data?

As a tester or developer you sometimes want, or need, to be sure that the only one manipulating the data during testing is you. This is because test results can be difficult or even impossible to predict when other testers are also active and manipulating parts of the data that your tests depend upon. A way to avoid this risk is to provide every test/tester with their own private test data set. DATPROF Subset can help you generate small, representative subsets, making it possible to deliver multiple test sets even in limited database space environments.

Having your own dedicated database and test data set for a specific test is a blessing when it comes to regression testing. You can reset the data in a database to a known pre-test state at any point in time without affecting other testers and without having to wait for other testers to finish their testing activities!

There are several strategies that can be used to reset the data in a database to its pre-test state. In this article two strategies are discussed, the first being the ‘rewind’ strategy and the other the ‘restore’ strategy.

The ‘rewind’ strategy

One of the strategies is to ‘rewind’ the user actions performed during the test. Imagine a situation in which you successfully Added customer ‘John Broke’ as part of your latest interface test of the Customer Registration screen. After the test you functionally Delete ‘John Broke’ using built-in application logic.  However, are you aware of any possible side-effects of your application? Maybe your application has a business rule implemented that keeps track of customers who are ‘deleted’ without ever having ordered something and are therefore blacklisted in a dark corner of your database.  The next time you want to add ‘John Broke’ as a customer your application will deny you because of the applications’ business rule enforcing the blacklisting during your ‘believed to be safe and sufficient’ attempt to reset (‘rewind’) the data to its pre-test state.

The application is now reacting differently to the test compared to the first time and you are just about to find out that functionally deleting a customer does not bring your test data back to the state it was in before the respective customer was added. If the customer’s registration in the blacklist table were the only side-effect to compensate for in order to get the test data set back into the pre-test state, you could try to technically remove that record from the table(s) the application is using to manage the blacklist information. Chances are, however, that the business logic and the data model are complex black boxes, or that you as a tester may regard them as such.

It’s also true to say that any rewind logic which you may design must also be failsafe. There has to be a way to recover the recovery process in the event of failure, otherwise your database will be left in an inconsistent state where the only recovery option is to restore from backup or regenerate the subset.  This is a vicious circle where the rewind option can take more time in planning, coding and implementation than the original application feature that is being tested!

The ‘restore’ strategy

Clearly, it is far safer and easier to use the ‘restore test data’ strategy which means removing all the post-test content from the database and subsequently restoring the pre-test data set from a backup copy. It goes without saying that this strategy benefits from having smaller test sets. In fact you and your colleague testers/developers will benefit, as well as your organization which ultimately pays the cost of testing and test data provisioning. You should also consider the difference it will make in terms of accumulated response delay times and the effects on data storage and data processing when compared with testing against full-size copies of the production environment.

The examples above demonstrate that ‘rewinding’ can be doubted as a safe strategy in regression testing. It is like having a 10 litre bucket that is filled with 9 litres of water in the pre-test state. During the test 2 litres of water are poured into the bucket. How do we best re-establish the pre-test state? By pouring 2 litres of water out of the bucket (rewind) or by emptying the bucket and refilling it with 9 litres of water (restore)?

Would you, having read this article, use or recommend the ‘rewind’ strategy for getting a test data set back to its pre-test state? If so, would it make much difference to you if the test database(s) were shared by multiple testers/developers or if every tester/developer (or sub-team) had their own test set in a private, dedicated database?