What a piece of work is a man, how noble in reason, how infinite in faculties, in form and moving how express and admirable, in action how like an angel, in apprehension how like a god! the beauty of the world, the paragon of animals—and yet, to me, what is this quintessence of dust?
Good question, Mr Shakespeare. In the end, man’s apprehension only goes so far, and that’s where we testers come in.
Everyone who’s ever worked with a more-than-trivial setup, that the very old piece of wisdom “Never change a running system” still applies, maybe even more so today, with all the interconnectivity and layers of complexity our systems have. Changing a live production environment is a challenge, moving it is a nightmare, moving it without moving the physical machines even more so.
You need a very detailed plan that covers all the interconnected servers, services, firewalls, databases. You need a contingency plan. You need to inform your customers about the downtime. You need test scripts. And you need testers. And, of course it all happens on the weekend and has to work perfectly on Monday.
It’s funny how time works differently when you migrate systems, it’s almost like it happens in a parallel universe where a minute is worth two of yours. As a tester, you can do nothing while you wait for the new environment to come online so you can do your job. There’s always one more port that needs to be rerouted or a file transfer that needs to finish. It can be very frustrating when “We might need you on the weekend, be prepared” becomes certainty – but not before the end of the week. You have all your scripts ready at hand, you join all the prep calls, but all you can do is wait.
It doesn’t matter if in the end all works out fine – everyone feels wrecked when it’s over. If your system has good testability and automated tests you can at least tell how confident you are that the new “prod” will behave itself. But if you don’t, for example because you do not get all the time you want for running your complete suite, as a tester you can only point out what you have verified working and what you don’t know – you are a walking disclaimer. Sure, you’re not the one making the final Go call. But as tester, you love facts, you want things to work – and you only trust them when you had the chance to try them.
So how do you prevent this situation? Have contingency plans not only for project failure, but also for the case that you don’t get the chance to run all your tests. Sometimes a release cannot be postponed, no matter how “disclaimatory” you get. Organize your planned efforts in tiers. “This is what I want to test. If I lose two days, I’ll only test this and ask Bob and Jane to help, although they don’t know the system so well. In the worst case, I’ll test these critical cases.” Really, have this ready before you start hearing about delays. Because you will hear about delays.
And: communicate early. Tell your people they will have to work on Saturday. Tell them it’s going to be painful. Don’t freak them out but give them what you have and why this is necessary. Get a commitment from the guys on other teams you plan on asking to help, get their line manager on board. If your IT crowd manages to pull a miracle and give you your testing time during the week after all, or the project is canceled, you can still roll back. It’s better to expect having to work late and be then be told it won’t be necessary than to speculate every day after each update call. And this is true no matter whether your people have a great “duty & responsibility” attitude or not, especially for testers who are typically lovers of facts and information.
Last but not least, be prepared yourself. Make sure every potential tester can access your issue tracking tool, the VPN, scripts, conferencing tools etc. on the weekend. Include non-techie instructions in your scripts for helpers from other departments. Have a single point of contact where executed scripts are sent. Keep track of the progress and any defects for the case the CTO calls from the airport between transfers, asking how’s it going.
And very last, don’t panic. Life’s too short.