I was recently an author of How to Succeed at Database and DBMS Migration and I often take calls on database migration. Migrations are becoming more frequent, for example prompted by moving to the cloud. DBMS Migration projects often overrun in time and cost and some fail completely. Inadequate investigation and planning in the early phases of the project is one of the most common reasons why migrations fail — a good example of the old project management adage “Failing to plan is planning to fail”. Here’s a useful tip that often comes up on my calls.
“Can this system be migrated?” is the Wrong Question
It may seem counter intuitive but the question “Can we migrate this database / DBMS?” is wrong. It causes the team to collectively share how confident they feel about proceeding, thus prompting a subjective assessment.
This question invites a wide range of cognitive biases and human frailties. These have been well documented in popular articles such as this one in The Atlantic magazine and books such as “Thinking, Fast and Slow”. People are typically overly optimistic in making estimates — to take just a few examples:
- Busy team members will not do a thorough investigation when pressed for time.
- They may be thinking of their immediate short-term problems. Carrying out a DBMS migration is tomorrows problem.
- A project leadership candidate may not want to look uncertain in front of peers.
- A team member may be unable to resist pressure from a forceful boss who’d like to pass good news up to their superiors.
Alarm bells should sound loud in your head when anybody asks you for a subjective judgement on the feasibility of a migration. If it all goes horribly wrong it will be you, the person giving the initial assessment, that will likely suffer the consequences.
The right question is “What will it cost to migrate?”
It is always possible to migrate a database system. Moving a database from one platform to another whilst keeping the same DBMS this can be relativity simple. However, if the team has to junk the old system and write the new one from scratch it is likely to be a large amount of effort. Rewriting the system completely is simply an extreme form of migration! — it may just take more effort.
Usually the team can convert or rewrite part of the system using the information about and documentation of the original system as a guide. Knowledge about the system will not be lost to the team.
Thus, the answer to the question “Can we migrate?” is always “yes”. What the organization really needs to know is what it will cost. By cost we mean how much time, effort, money and resources it will take. It is then possible for management to compare the estimate of costs to the amount of resources available and check that the project is feasible.
What you need to know
Asking about the cost will focus the project on the right things. To answer the question it is necessary for the team to understand all of the:
- Data items in the system, including archives, backups, disaster recovery sites — every piece of data. Any piece of data skipped over is an opportunity for a later surprise.
- Different types of data and how the developers will convert them and transport them to their new home.
- Code — system and application. For every piece of code we need to understand its type and the method needed to translate it to the new system. In DBMS migration the team must also understand the translations between the different DBMS commands. The team needs to consider all the different languages used, control statements, configuration files etc. Thus, there are more types of code than just the programs contained in the system.
- Testing. The designers need to ensure that all of the above can be tested and then delivered safely into production. This usually entails a production line of steps that each piece of code and data should move through. It should include system, performance and user acceptance testing.
- Other elements that require effort by the team to set up, move, or test. For example, documentation, operational procedures (like backup / recovery and HA/DR) — plus training courses.
It does not matter whether the conversion work is automated, semi-automated or manual. To provide a sensible idea of the resources needed, the team needs to understand all of the above, and more. The estimators can’t be certain of any estimate if they haven’t taken into account all the data and all the code. If the technical staff hasn’t checked for the different types of data and code and understood how to convert each type then they cannot know the total costs. Likewise, any assurance of feasibility is worthless if they don’t understand the testing process, and any special tools or code needed to carry it out.
Remove the Subjectivity
Estimating the cost focuses the team on areas of uncertainty. It reduces the emotion and subjectivity. The team will either know the scope of the migration or it won’t. The technical staff will have have identified all the methods to be used to convert all the types of code, or they won’t. The test experts will have designed the test harness and estimated its development effort, or they won’t. These things are easy for the the team to check by asking if they can explain them to each other. To be sure, there is still the possibility of errors in estimation but it will have been reduced. The discussion is now around a specific set of data and code objects and the work the team needs to do for each.
The team should not agonize over this. The only useful thing for the estimators to do is to plug any gaps in understanding. No law says the team has to do this in one pass. The estimators and team should do multiple passes, and do a “sanity check” each time. After a few passes the team will find that the estimates will stabilise and become increasingly explainable and defensible.
As we can see, despite their being complex and risky projects, DBMS migrations can be accomplished successfully and with a positive payback . It is all in the planning.