A Bird’s Eye View of Data Migration

Data migration can be incredibly daunting. Finding a home for every field, pixel, and document in a new system is like moving between libraries that have completely different classification systems. Or like helping an elderly relative move out of their decades-long home and downsize into a new apartment. Often it feels like being caught up in the middle of birds in flight, wings and feathers flying on all sides.

We’re going through this process at work, and a few high-level principles have helped guide our move from a decades-old, highly customized IBM system into a cloud-based ERP.

• Rationalize the heck out of the data! Identify business needs and scope accordingly. How many years of records do you really need to migrate? Consider accounting and legal requirements: how many years will the IRS require? Do products have warranties or safety documentation that need to come over? Activity is another great yardstick: what’s the typical length of customer relationships, what are products’ turn rates, and how far back does sales history need to extend? Finally, what current inventory and future commitments need to be moved over?
  
• Prune, baby, prune. What can you live without? Which tables or fields were created to meet a need that’s met in the new system? What fields in the new system are irrelevant to your business and can be hidden? If you leave legacy data out of initial testing, does anyone notice it’s not there? Can the business live without it? Moving systems is a rare chance to purge and simplify: take full advantage of it.
  
• Clean all the things and govern! Again, moving house is a chance to clean the scoped data set you’re migrating, which naturally requires governance. If, for example, you have delivery notes in the address fields, what business need gave rise to that? Was there no dedicated field in the old system, was it the easiest way to communicate with the drivers, etc? Pivoting to the new system, can it be handled differently, and how are you going to keep people from continuing in their old ways? Where can you in build guardrails to keep your new datasets clean?

The migration allows a business to take stock and very deliberately set a future course. The shuffle of field mapping, setting up SSIS packets, and long discussions about how to use field X can obscure the grandness of a flock winging south in formation. But every now and then, you catch a glimpse of the ground speeding by below, and realize you’re moving very fast indeed.