You are here: Home Blog Software Development A Little Data Cleansing Goes a Long Way
Tuesday, 20 November 2012 21:35

A Little Data Cleansing Goes a Long Way

Written by 

We can easily recognize the effort, resources, and technology that went into building both the Concorde and Space Shuttle Columbia.  However, not everyone can understand and appreciate the little things that are essential for the success of the mission.

Those of us who are in software and information technology understand how projects are done, what makes them succeed, and the resources (time, effort, and money) that goes into them.   Project components are frequently complex and can consume major resources, so we tend to focus on these larger aspects when we review what is critical for the success of the project.  However, focusing too much on the big tasks makes it easy to overlook tasks or components that appear simple, inexpensive, or less glamorous.


Take, for example, cleaning the runway at an airport to get rid of dirt, left over debris, and garbage. This is not definitely a glamorous thing nor is it considered as critical as inspecting the plane before taking off. As you can see from the example of the Concorde accident in January 2002, there was a huge impact in leaving a small strip of metal that fell from another plane.  The consequences of seemingly small and insignificant factors on the success and survival of major missions cannot be ignored.

Another example is the Columbia disaster of 2003. When you take a look at the processes deemed critical for launch, the foam on the wings did not seem like a major element deserving as much attention as other larger issues.  But the tragic story paints a different picture.

As CEO of an IT vendor for well-known companies, I’ve seen many situations where small details that were ignored made a huge impact on the outcome.  I am aware of a customer who had an application running on IBM mainframe. We had the opportunity to work on a system to replace the 20 plus year old legacy application. The new solution built on state-of-the-art technology is superior not only in architecture, but clearly saves money by optimizing work flow and offers flexibility to adapt to future changes.

The scrutiny was on many aspects of the product, including response times, performance testing, and so on. The implementation date was postponed couple of times to make the customer comfortable. During all of this time, the focus was on the vendor to deliver the product – did we get all the features we need, did we test enough to verify we got  all we wanted, and did we try to break the system to ensure that it endures the rigors of usage in the field. One thing that went ignored was the customer’s task of ensuring that clean and proper data was migrated from the old system to the new system. This could have been done at a cost of just a little more than the resources they were investing from their internal teams. The end result was a scary first 2 weeks of deployment when cleanup from this fallout proved far more expensive than the amount they thought they saved by skipping this small step. 

I always compare this experience with that of another organization which was consolidating numerous legacy applications into a new system, costing them more than $300M to build. The CIO and the management understood the significance of planning, and the value of the non-glamorous components. They recognized the value of the data cleansing and data conversion. They went ahead and started procuring resources to contract for completing the data analysis and cleansing work two years before the anticipated deadline for the delivery of the project. They spent approximately $20M in data analysis and cleanup, ensuring that there were no major issues for this project.

There are many options to create an application serving a particular business need, including but not limited to building a custom application, buying and customizing an ER or off-the-shelf-package, modernizing a legacy application, and reengineering existing systems. In almost all of the cases there will be non-glamorous and seemingly not crucial components that draw all the attention. These could be things like data analysis, data cleansing, and data migration.  It is important that businesses don’t lose sight of these small details to avoid failure.

Data analysis, data cleansing, and data migration may appear to be noncritical and not glamorous. But these are very much essential for the smooth deployment of many IT solutions when replacing one system with the other. Whether you do it internally or use external assistance, please remember the Concorde and Space Shuttle Columbia as you plan for your next big project.

-Srini Madala, SoftSol Founder and Chairman

Read 21247 times Last modified on Wednesday, 27 February 2013 06:45

Leave a comment

Make sure you enter the (*) required information where indicated. HTML code is not allowed.

Contact Us

SoftSol Inc.
42808 Christy Street,
Suite 100,Fremont,
CA 94538

Phone: 510-824-2000
Fax: 510-217-3461
E-mail: This email address is being protected from spambots. You need JavaScript enabled to view it.

Find Us on Social Media

FacebookTwitterLinkedInRSS Feed