I have an issue in both the projects that I’m currently involved in and it has to do with debugging issues in previous versions of the application. In essence I cannot run an older version the application as my current database schema is newer than the model objects that map to it. Essentially I need a way to rollback a few database migration scripts so that I can run the application, or alternatively recreate a fresh database. This later solution should always be available, but as of now neither of them can be done. If the version I’m trying to run is the current production’s version, fortunately I can use a SQL dump of the production database to fix my error and allow the application to run.
So, I’ve been thinking of how to solve this, and this evenutally led me to thinking also about how to handle acceptance tests, which share the same problem. Basically both database migration scripts (which also allow to rollback the migration) and acceptance tests shouldn’t exist in the same repository as the rest of the application. The reason is that if I need to rollback the database schema to a previous version, I need the newer migration scripts to perform the rollback (as well as figure out at a particular application version what version the database should be in). Similarly, and with a different importance, I might want to test if a previous version of the application adheres to a newer (either newly created or modified) acceptance test, so here also I must keep it on a different history line than the application itself.
I need to explore a complete solution for the database migrations repostories, and eventually I’ll try to blog about my thoughts (and possibly solution(s)) about that when I get a change, but for now I’ll concentrate on acceptance tests.
First it must be clear what is meant by acceptance tests. In plain terms they are the tests that should try to run a close-to-production version of the application, excluding external dependencies and effects (such as 3rd party remote services and emailing real email accounts), and they should test the features of the application. According to the semantic versioning specification (SemVer) features should only be added or backward-compatibly modified for minor releases (x.Y.z) and backward-incompatibly modified or removed for major releases (X.y.z). Backward-incompatible changes to features change the acceptance tests, so it is important that when a feature is changed (including removed) that the acceptance tests for the previous major release are maintained and kept separate from the (new) acceptance tests for the modified feature. This way the acceptance tests for a previous major release are still available and runnable on the older version.
Whilst both thinking about this on my own and trying to see if there are any current solutions described and/or uses I came upon the Mozmill tests which are the functional tests for Firefox. The setup used there is of a seperate repository which uses seperate branches for the various major releases, so they have a mozilla2.0 branch for Firefox 4.0.x, mozilla1.9.2 branch for Firefox 3.6.x, mozilla1.9.1 branch for Firefox 3.5.x, etc. This way all acceptance tests for each major and minor release are put in their specific branch and in all branches of the subsequent releases. I really like this solution because it very simply solves all the issues I can think of, and I would suggest it to anyone who is also looking for a solution to this problem.
If, however, I’ve missed something or there are any suggestions or thoughts about this, I would gladly hear what you think about it, so please leave a comment below!
dwijnand::blog 