Complex data migrations done well

Published on: August 11, 2020

When organizations migrate to a new software application, usually a lot of attention goes to system selection and configuration. What typically generates less enthusiasm is making sure that the data that needs to populate the new system is fit for purpose. And yet, data migrations are critical for business success. Therefore, they require focused attention and a structured approach, especially in complex, asset-intensive markets.

In today’s data-driven business world, data quality plays a crucial role in how businesses perform and how decisions are made. When an organization’s data system changes, it’s not just a matter of copying and pasting data from one system into the other. Especially in organizations that manage a large asset database, like utilities and manufacturing companies, data migration requires careful planning and a staged approach. More often than not, data migrations are projects of many months.

Data migration strategies

Say you run a small book shop, and you want to change over to a new inventory system. Chances are that either you or someone in your team will have to transfer all the current book references from the old system into the new system. Typically, this data migration will require your system to be down for a while and you will perform the migration in one go, probably over the weekend or on a holiday.

As long as you are running small numbers of data, this can work fine. However, as you are pounding away data into your Excel file over the weekend, you will probably not be able to predict whether your data migration will be 100% successful. And if something turns out to go wrong at the end of the weekend, you may have a hard time finding out where you made a data entry mistake.

From the moment you are working with large, complex data sets, you need a data migration strategy that involves less risk. Often the migration involves several databases, the data of which are combined and integrated into a single new database. A professional data migration process will migrate data in stages, not in one shot, and this will happen through a repeatable, predictable process. By breaking the migration down into smaller parts, it is easier for the organization to assess the success of the migration, and you will be able to learn valuable lessons along the way to improve your migration strategy in later stages.

But even with a staged approach, data migrations may result in data mismatches that will cost time and effort to solve. To avoid this as much as possible, data migrations can also be performed while both the old and the new system are running in parallel. Running two systems in parallel during the migration may be more complex and more costly, but when you manage data that is a bit more life- or business-critical than a book shop, this may be the way to go.

The migration chain

A data migration project includes both the description and development of functional requirements (what do you want to do with the data) and the actual data migration (moving the data from a to b). At Kapernikov, we work closely together with data users, in order to understand their functional requirements in depth and to configure the new system in line with those requirements. Plus we help our customers through a data migration process that is highly automated, and that allows us to load complete data sets into the new system in a predictable and repeatable way.

Our structured approach to data migration includes:

Data extraction: We automate the data extraction from the source system and perform data cleansing on the source data to improve quality. We collaborate on correcting data to make sure that it will be better adapted to the requirements of the target system.
Adding extra data: Sometimes, data that was not included in the old system will need to be added to the new system.
Modelling the data: elements can be adjusted or added, so as to offer additional information, establish links, enhance user-friendliness, enable future data integrations by aligning with industry standards. This is about taking advantage of the new system’s capabilities.
Data mapping: In this stage, we map data fields from a source system to their related target fields.
Reporting on progress: We log and visualize the migration.
Conversion algorithm: Based on the previous stage, we are able to determine detailed mapping specifications or rules for moving source data to the target system. We allow exceptions to our rules, but the goal is to keep the number of exceptions as low as possible.
We load data into the new system according to the conversion algorithm.

This data migration chain is used across the entire data migration process – during development, test phase, and user acceptance testing – and it is based on real data sets. The goal is to load data into the new system as soon as possible, as part of an agile approach.

Common data migration mistakes you want to avoid

At Kapernikov, we have had the opportunity to support data migrations with some of the most important manufacturers and utility providers in Belgium. As we have learned along the way, a complex data migration project has its own set of challenges. Here are some of the most classic ones.

Inefficient planning: Be it because you started too late or your data migration was not a priority, inefficient planning is a recipe for disaster. And when you choose a non-staged approach, a data migration late in the process puts additional stress on the planning and data issues can lead to costly downtime.
Too few resources: Data migration projects can be complex, especially when multiple systems need to run in parallel. Having juniors manage the data migration project for one day in a week is the surest way to failure.
Acceptance testing with insufficient data: Insufficient data will result in an irrelevant acceptance test, which in turn may lead to serious performance or other issues when going live.
Undiscovered source data quality issues: Many organizations have no idea about the quality of their data. Assuming that your data quality is fine, just because you use it every day, can be a wrong bet. More often than not, the devil is in the details and you risk derailing your data migration project at the end.
No dry run: Performing a dry run before executing the data migration in production is an efficient way to mitigate risk and avoid failure of the data migration at the critical moment. A dry run performs all the same operations as a production run, but sets up the target objects in an isolated manner so as not to interfere with normal business activities.

Worry-free data migration

Are you planning a large migration of your asset data? Then don’t leave it up to chance. At Kapernikov, we can help your organization through all steps of the data migration process, and we reduce the risk of downtime, data issues or other unexpected costs, to a minimum.