Making a success of CRM data migrations

January 21, 2018January 21, 2018 Mark Payne

One of the most critical aspects of any CRM implementation is the successful migration of data from legacy systems to your shiny new software. Unfortunately, going live to your users with poor data is a sure-fire way of denting their confidence in the system from day one and hence decreasing system utilisation. Often, the process behind a successful data migration is poorly understood by anyone other than the system implementer (hopefully they at least do understand it).

This blog post aims to explain what I believe are the critical steps to ensure a successful data migration so that you can more easily judge any proposal being put to you and ask informed questions of your implementation partner.

In my opinion, the most successful data migrations consist of 6 steps:

1. Initial data quality assessment.
2. Data mapping.
3. Development of migration scripts.
4. Test migration.
5. Data cleanse.
6. Live migration.

That may seem like a lot of steps to simply import some data into a CRM database but each one has its own purpose and will contribute to the overall success of the project as a whole. I will now look at each step in turn.

Initial Data Quality Assessment

The purpose of the initial data quality assessment is to take an honest look at the source data you would like to import into the new system and decide what is good and what is bad. Ideally this should be a joint exercise between yourself and your system implementer and it is often completed during the early discovery or design phases of the project. The output of the data quality assessment should be as follows:

1. Which data sources (if any) are in a good enough state to even be considered as candidates for migration to the new CRM?
2. Of the data sources that are suitable for migration, what can be done to make them as clean and accurate as possible before they are migrated?

These decisions will feed into any business or functional requirements document and will allow your partner to accurately plan the required data migration activity. Of course, you should always limit the sources you are looking to bring in to those that will directly feed into the business processes that the CRM system is due to support.

Data Mapping

During the data mapping exercise, your implementation partner should produce a document that details each field in the data sources you have identified for migration, where that field maps to in the new system and any data transformations that need to take place. This can often be a quite technical document but there are a number of items that you should pay close attention to:

1. Make sure that every field in your source(s) has been covered (even if that field is not due to be brought in to the new system). It is important to get a full picture of the data available so you can ensure that the choices made about what to migrate and what not to are correct.
2. Always look for any fields that are documented as “not mapped” or “not to be migrated”. Make sure you are happy that this data will not appear in the new system.
3. Make sure that any mandatory fields in the new system that will not be populated by the data migration have sensible defaults applied. For example, if you are importing account records, and “Account Type” is a mandatory field not present in the data you are migrating then ensure you know what that field will be defaulted to.
4. Make sure that any data transformations specified make sense.
5. Make sure that some way of easily identifying the migrated data in the new system is specified. This will ensure that the data can easily be removed between the test migration and the live one.
6. Make sure that some way of referencing the data in the new system back to the source data is included. If there is a unique identifier in the source data for each record being migrated, make sure it is copied into the new system. This will allow records to be updated from the old data post-migration, if necessary.

This data mapping document should form the basis for all subsequent migration effort so it is important to get it right. As such, you should plan to dedicate what may seem a disproportionate amount of time to reviewing it and talking it through with your implementation partner.

Development of data migration scripts

During this stage of the data migration process your implementation partner will get on with the technical exercise of writing scripts or code to import the data as specified in the data mapping document. This work will be largely invisible to you but, like any build phase, you should expect regular updates on progress and also respond in a timely manner to any questions put to you.

You can use this time to prepare the source data in any way that your system implementer has agreed with you ready for the initial test migration.

Test Migration

Once an initial version of the migration script(s) is ready your implementation partner should perform a test migration of all source data into the development version of your new CRM system. This exercise allows you to see what the migrated data will actually look like in the new system and will verify the decisions made during the initial data quality assessment and data mapping exercises. It will also allow the system implementer to see how long the scripts take to run so that they can plan the live migration accurately.

Your partner should also be able to provide you with a list of any data issues or anomalies that they found during the test migration.

Once the test data is in your development system you should take the opportunity to:

1. Verify that all the data you are expecting to see is available.
2. Verify that the data looks accurate. At the very least you should dip-test some records and compare them to the old versions in your data sources.
3. Use any tools available to you in the new CRM system to assess the quality of migrated data (e.g. duplicates, missing data etc.).

The output of the test migration is normally two-fold:

1. An amendment to the data mapping document in reaction to how the data actually looks in the new CRM system.
2. A list of data cleansing tasks that you or your partner need to undertake, prior to go-live, to ensure that the best quality data possible gets into your new database.

If necessary, multiple test migrations can be performed until you are happy with the data in the system.

Data Cleanse

Once you have your list of data cleansing tasks you (or your partner) will need to assign resources to perform them. More often than not these exercises are manual and quite labour intensive as there is nothing better than the human eye at tidying up data. You should not underestimate the effort required (or indeed the impact of not carrying out this step). As I said at the beginning of this article, clean data at go-live is essential to the success of the system.

It should be noted that there may be data cleansing tasks that are better performed post migration using the tools available to you in your new CRM system. Don’t be afraid to make use of these tools but do make sure that the tasks are scheduled to be completed very soon after go-live!

Live Migration

The final step in the whole process is the live data migration. All live data migrations should happen immediately before the system goes live and will involve down-time on your old systems (if they are your sources of data). The amount of down-time required will depend entirely on the length of time required to run the migration scripts. If the amount of down-time your partner is requesting is unacceptable then you can either choose to have them run it out of hours (e.g. over a weekend) or have the migration performed in stages with only the essential data being migrated prior to go-live.

In general terms, all live data migrations will follow this pattern:

1. Test data is removed from the new system.
2. Work ceases on the old systems that form the migration data sources.
3. Migration scripts are run.
4. Live data is verified.
5. System goes live.

The two stages above to pay particular attention to are number 2 and number 4.

Once your partner tells you to stop using your old systems make sure that your entire user base knows to do exactly that. Failure to do so can mean that last minute updates are lost and/or the users are unable to verify the accuracy of the data in the new system.

When your partner asks you to verify the live data make sure you have a written set of criteria that will allow you to quickly check that the data in the new system is both complete and accurate. For example, you may wish to check the number of migrated accounts in the new system against the number in the source data.

Conclusion

Data migrations are a delicate and often time-consuming part of any CRM system implementation. This is time well spent as most of the value in your new CRM system derives directly from the quality of data it holds. In my experience, by following the steps outlined above and working closely with your implementation partner at all stages of the process you should be able to get the data that you need to your users from day one and hence give them no reason not to use the system!

One final thought, do not discount the possibility of manually keying any data into the new system. It may actually be cheaper and easier to employ some temps to enter the data rather than going through the whole data migration process.