Customer data integration (CDI) projects demand transformation of people, process and technology. If just one area fails the entire project will be at risk.

Here are common pitfalls that may turn a CDI initiative into failure, according to SearchDataManagement.com:

• Data model misfortunes
  Vendors usually promote those data models were designed to support their applications. They can’t offer flexibility around the data model. However, a predetermined data model, designed by an application vendor, can be a big reason for CDI failure.  Companies should look carefully at various data model approaches and options and choose those answering their needs.
• Overarching architecture and technology issues
  While opinions again vary on the right technical and architectural approach, it’s clear that there is no “one size fits all” approach to CDI. A company’s existing systems, potential longer-term changes and specific business requirements all play a part in determining the correct technological approach for a given project. Issues of CDI system architecture, performance and scalability are all very important to consider up front, and flexibility is critical.
• No plan (or budget) for long-term maintenance and extensibility
  CDI is not a one-shot deal. It’s an ongoing project. Early adopters of CDI, many of whom built their own in-house systems, are now looking toward commercial products. They have found out about the long-term costs and requirements of maintaining custom systems.
• Lack of user adoption
  Education and training are extremely important parts of CDI implementation and must be a primary part of a CDI plan, not an afterthought. Many companies using CDI successfully recommended developing a corporate-wide marketing plan for a CDI project and associated data governance and data stewardship initiatives.
• Not addressing data quality, governance and stewardship issues
  CDI projects without an associated enterprise-wide focus on data stewardship and data governance will fail, many attendees said, because the source of the data problems might not be addressed. Worse yet, if users discover data problems in the context of a CDI implementation and start to distrust the data in the system, the whole CDI project can fail due to skepticism leading to lack of user adoption.
• Politics, pure and simple
  Underlying many CDI-challenge discussions was the issue of corporate politics. By definition, CDI projects can touch almost every department in an organization. Whether it’s the lack of enterprise-wide support, unrealistic timelines, inadequate budgets or data ownership issues, CDI project leaders must be social champions, as well as architectural experts. Data management can be a touchy issue, according to many attendees. Various divisions may feel that they own the data in their own systems and may be reticent to allow another system to access — much less change — what they consider to be their critical information.

At this stage several important questions should be asked and answered about specifications and requirements. By the end of this phase you should have a thorough specification of how the source and target objects will be mapped to pass the results to a developer for implementation in a data migration tool. You also should have a firm interface design, data quality specifications, the firm idea of what technology will be required in the production environment, and, finally, service level agreements.

Here are the details:

• Have you created a detailed mapping design specification?
  Note that there is no immediate progress into build following landscape analysis. It is far more cost-effective to map out the migration using specifications as opposed to coding which can prove expensive and more complex to re-design if issues are discovered.
• Have you created an interface design specification?
  A firm design is needed for any interface designs that are required to extract the data from your legacy systems or to load the data into the target systems. For example, some migrations require change data capture functionality so this needs to be designed and prototyped during this phase.
• Have you created a data quality management specification?
  This will define how you plan to manage the various data quality issues discovered during the landscape analysis phase. These may fall into certain categories such as:
  •     Ignore
  •     Cleanse in source
  •     Cleanse in staging process
  •     Cleanse in-flight using coding logic
  •     Cleanse on target
• Have you defined your production hardware requirements?
  The volumetrics and interface throughput performance should be known so you should be able to specify the appropriate equipment, RAID configurations, operating system etc.
• Have you agreed the service level agreements for the migration?
  At this phase it is advisable to agree with the business sponsors what your migration will deliver, by when and to what quality.
  Quality, cost and time are variables that need to be agreed upon prior to the build phase so ensure that your sponsors are aware of the design limitations of the migration and exactly what that will mean to the business services they plan to launch on the target platform.

Why enterprise-wide data quality initiative often turns into a total disappointment? According to Forrester’s report “A Truism For Trusted Data: Think Big, Start Small,” it happens because of managers’ ambitions to implement an enterprise-wide system for trusted data all at once.

However, experts from Forrester recommend thinking global, but starting small. In other words, they advice consider a bottom-up approach that defines quantitative and qualitative ROI for only those few select functional organizations that can best articulate and measure the business impact poor quality data has on processes.

Here is a short overview of the steps, Forrester recommends, to effectively implement data quality initiatives:

1. First of all, those responsible should start from defining what they mean by “data quality” and “trusted data”. As Jill Dyché once mentioned, it’s time to understand the following seldom-understood truth: That there are different levels of “acceptability” for data. And, according to her, the key is to understand company’s business requirements and then drill them down to data requirements. That will tell conclusively what good enough for the company really is.

Forrester defines “data quality”  as “data used by business stakeholders to support their processes, decisions, or regulatory requirements with no reservations as to the data’s relevance, accuracy, integrity, and other previously agreed upon definitions of quality.”

Forrester reminds that data quality must come directly from business stakeholders, for they are those who understand business requirements of the company, and thus may set the standards for company’s data quality.

2. Then, Forrester insists on building a business case that starts small.

“Scoping and prioritization based on the business processes within the organization that are most critically affected by poor data quality is the key to defining the business case that will get your trusted data initiative off the ground,” say experts.

3. As word of the value of the data quality project gets around, other organizations, such as those responsible for order management and fulfillment, may also want to implement data quality improvements within their environments.

“Eventually, the tide will turn and these business stakeholders will sign up and support an enterprise-class solution to solving their data quality problems, but for that to happen, value must be demonstrated,” Forrester concludes.

Actually, this phase is more a kind of a sequel of the previous one, so it would be better called Phase 2.2. You know, now I understand why so many enterprise data migration initiatives go wrong, you need to bear in mind soooo many things. The process demands constant interaction of all the people involved… Okay, let’s continue checking the data migration to-do list.

• Have you determined high-level volumetrics and created a high-level scoping report?
  Ensure that you fully assess the scope and volume of data to be migrated.Focus on pruning historical and redundant data. Create a final scoping report detailing what will be in scope for the migration and get the business to sign this off.
• Has the risk management process been shared with the team and have they updated the risk register?
  Create a simple online form where anyone can add risks during their analysis, you can also filter them out later but for now we need to gather as many as possible and see where any major issues are coming from.
• Have you created a data quality management process and impact report?
  If you’ve been following our online coaching calls you will know that without a robust data quality rules management process your project will almost certainly fail or experience delays. Understand the concept of data quality rules discovery, management and resolution so you deliver a migration that is fit for purpose. The data quality process is not a one-stop effort, it will continue throughout the project but at this phase we are concerned with discovering the impact of the data so decisions can be made that could affect project timescales, deliverables, budget, resourcing etc.
• Have you created and shared a first-cut system retirement strategy?
  Now is the time to begin warming up the business to the fact that their beloved systems will be decommissioned post-migration. Ensure that they are briefed on the aims of the project and start the process of discovering what is required to terminate the legacy systems. Better to approach this now than to leave it until later in the project when politics may prevent progress.
• Have you created conceptual/logical/physical and common models?
  These models are incredibly important for communicating and defining the structure of the legacy and target environments.
• Have you refined your project estimates?
  Most projects start with some vague notion of how long each phase will take. Use your landscape analysis phase to determine the likely timescales based on data quality, complexity, resources available, technology constraints and a host of other factors that will help you determine how to estimate the project timelines.

Continuing my previous post, here are the questions to ask oneself at the stage of data migration project initiation:

• Have you created a stakeholder communication plan and stakeholder register?
  During this phase you need to formalize how each stakeholder will be informed.
• Have you tweaked and published your project policies?
  Now is the time to get your policies completed and circulated across the team and new recruits. Any policies that define how the business will be involved during the project also need to be circulated and signed off.
• Have you set up your project collaboration platform?
  This should ideally have been created before project initiation but if it hasn’t now is the time to get it in place.
• Have you created your standard project documents?
  During this phase you must create your typical project documentation such as risk register, issue register, acceptance criteria, project controls, job descriptions, project progress report, change management report, RACI etc. They do not need to be complete but they do need to be formalized with a process that everyone is aware of.
• Have you defined and formalized your 3rd Party supplier agreements and requirements?
  Project initiation is a great starting point to determine what additional expertise is required.
• Have you scheduled your next phase tasks adequately?
  At this phase you should be meticulously planning your next phase activities so ensure that the business and IT communities are aware of the workshops they will be involved in.
• Have you resolved any security issues and gained approved access to the legacy datasets?
  Get approvals from security representatives (before this phase if possible) and consult with IT on how you will be able to analyze the legacy and source systems without impacting the business.
• Have you defined the hardware and software requirements for the later phases?
  What machines will the team run on? What software will they need? What licenses will you require at each phase? Model re-engineering tools? Data quality profiling tools? Data cleansing tools? Project management software? Presentation software? Reporting software? Issue tracking software? ETL tools? It can often take weeks to procure this kind of equipment so you ideally need to have done this even before project initiation.

Data migration is a complicated and complex process, so it is need to identify the risks and activities well in advance. A proper planning of the process is an essential part of a data migration project.  A good structured plan may help not to overlook some points, which may prove to be crucial for the whole project, and thus lead to project delay or closure.
I’ve recently found a very good and detailed set of questions one should consider when embarking on a data migration project by Dylan Jones from Data Migration Pro Journal, and I’d like to share it.

Here are the items one should check on a pre-migration phase:

• Have you assessed the viability of your migration with a pre-migration impact assessment?
  It is advisable in a pre-migration level to verify the cost and viability of the migration, including its terms (how long it will take), the choice of technology it will require and provisions for dangers that may lie ahead.
• Have you based project estimates on guesswork or a more accurate assessment?
  Provide accurate analysis of cost and resource requirements so if you have tight deadlines, a complex migration and limited resources make sure you perform a migration impact assessment asap.
• Have you made the business and IT communities aware of their involvement?
  It makes sense to inform the relevant data stakeholders and technical teams of their forthcoming commitments before the migration kicks off. There are numerous aspects of the migration that require business sign-off and commitment. Make sure everyone understands and agrees to what their involvement will be.
• Have you formally agreed the security restrictions for your project?
  Obtain a formal agreement from the relevant security governance teams in advance. Simply putting your head in the sand and hoping you won’t get caught out is unprofessional and highly risky given the recent loss of data in many organizations.
• Have you identified your key project resources and when they are required?
  The essence is to understand the key migration activities and dependencies then plan to have the right resources available when required.
• Have you determined the optimal project delivery structure?
  Agile, iterative project planning with highly focused delivery drops is far more effective than a classic waterfall design, so ensure that your overall plan is flexible enough to cope with the likely change events that will occur. In addition, it’s wise to provide for delay occurrence due to contingencies.
• Do you have a well defined set of job descriptions so each member will understand their roles?
  Have an accurate set of tasks and responsibilities for each member of the team involved in the project. Clear understanding of what your resources need to accomplish will help you be fully prepared for the project initiation phase.
• Have you created the appropriate training documentation and designed a training plan?
  Data migration projects typically require a lot of additional tools and project support platforms to function smoothly. Ensure that all your training materials and education tools are tested and in place prior to project inception.
• Do you have a configuration management policy and software in place?
  Data migration projects create a lot of resource materials. Profiling results, data quality issues, mapping specifications, interface specifications… Ensure that you have a well defined and tested configuration management approach in place before project inception.
• Have you planned for a secure, collaborative working environment to be in place?
  If your project is likely to involve 3rd parties and cross-organizational support it pays to use a dedicated product for managing all the communications, materials, planning and coordination on the project.
• Have you created an agreed set of data migration policy documents?
  There are a multitude of different policies required for a typical migration to run smoothly, it pays to agree these in advance of the migration so that the project initiation phase runs effortlessly.

Last week SourceForge started accepting nominates for their fourth annual Community Choice Awards to recognize projects and their authors who have excelled in the following software categories:

Best Project
Best Project for the Enterprise
Best Project for Gamers
Best Tool or Utility for SysAdmins
Best Visual Design
Best Tool or Utility for Developers
Best Commercial Open Source Project
Best Project for Academia
Best Project for Multimedia
Best Project for Government
Most Likely to Change the Way You Do Everything
Best New Project

Users may nominate the projects they think are the best until May 29th, and then ten projects with the most nominations in each category will become finalists. The winners will be announced on SourceForge.net on or around July 23, 2009.

 Apatar will also take part in the event. So, if you like the project, click the button below and nominate us!

Among the major responsibilities of IT management, ensuring that the business has access to reliable information is, probably, the most critical. This is actually a very challenging goal for many companies because the data needed to support business decision making is often inconsistent, redundant and of poor quality.

Data integration is  required at the stage of creating a companywide environment of trusted information. This process requires a well-thought out architectural approach that will provide information about the business as a service to everyone who needs it. This architectural approach will typically require technology for ETL (extract-transform-load), quality management, the creation of a metadata layer, and a strategy for master data management (MDM).

Companies can often identify why data integration is required, but then fall short on implementing the technology in a way that maximizes the benefit to the business. It can be very challenging for companies to manage the data integration process successfully.

Marcia Kaufman, software industry analyst,  have gathered and overviewed in her blog ten common mistakes that should be avoided when planning for data integration,

1. Following a “fire-drill” approach to data integration. It is short sighted to use ETL technology as a tool to solve a one-time data integration problem rather than using this technology as part of a comprehensive approach to information management.
2. Not thinking about data as a shared and reusable resource. It is easier to budget based on getting a single task done. However, it is much more efficient and cost effective to be able to reuse data resources once the second, third and future projects are initiated.
3. Thinking tactically about data integration and missing out on opportunities to improve business process. Companies often implement data integration technology to eliminate time-consuming and labor-intensive processes that have been required to gain a consolidated view across business units. However, it is a mistake to focus on reducing head count and saving time in the data integration process, without also considering a broader strategic view towards improving overall business processes.
4. Not establishing an architectural framework with the capability of providing reusable information services. Once the data is decoupled from the business application, you need to develop a methodology that supports reuse so the data can be shared in different ways as needed. The information as a service approach is designed to ensure that business services are able to consume and deliver the data they need in a trusted, controlled, consistent, and flexible way across the enterprise.
5. Using software code to adjust for differences in definitions about customers, products, and other data types on a one-off project basis. In order to deliver information as a service, there needs to be repeatable way to manage complex processes without the expense and time required for recoding. This can be accomplished with the support of a metadata infrastructure.
6. Integrating data without placing a high priority on data quality. It is critically important that companies establish processes to cleanse and correct data as part of the overall data integration process. Creating standardized and consistent information will ensure that business users are more confident about business information and in a better position to grow the business and remain competitive.
7. Not creating a standardized way to handle data that is common to the various disparate IT systems and business groups. Companies need to understand the commonalities across different data types. This can best be achieved by developing a master data management (MDM) strategy to serve as the system of record for the consuming systems and applications.
8. The technical integration team and the business experts do not communicate effectively. There needs to be a shared and common language describing business processes to enhance communication between business and IT management. The business is more likely to have good quality information they can count on if the IT and the business establish an efficient process for sharing knowledge and requirements.
9. Business owners are reluctant to give up ownership of data. In order to gain the efficiency and accuracy in the data integration process, it is important to establish a consensus among the various data owners regarding data terminology and definitions, and there needs to be a clear understanding of the data lineage and who is responsible for these data over time. This often requires a significant cultural change because individual business experts often have a long history of managing data for their line of business or department as if it was a stand-alone entity. Companies need to find a way to balance the need for individual business experts to maintain control over their own data with the need for centralized management of data within a metadata environment.
10. Trying to do too much in one project. When data is integrated across departmental data silos, previously inaccessible data becomes available to business users. Companies can take on projects that would have been impossible before because of the enormous amounts of hand coding and manual data collection that would have been required. However, these benefits can be lost if companies try to tackle too much at once. Enterprise-wide information management projects must be approached in an incremental way so that there is time to evaluate and improve data quality, understand the needs of the business, and establish repeatable methodologies and processes.

Today MDM becomes trendy, though not everyone fully understands what it really means for an organization. However, integrated MDM plays an important role in reducing compliance risk due to a lack of data transparency and trust, especially in the financial industries. In general, MDM aims to provide consistent, comprehensive core information across an enterprise, enables companies to realize internal efficiencies by reducing the cost and complexity of processes that use master data.

I’ve recently found some ways, how MDM can reduce costs:

• MDM centralizes common information, thus simplifies expensive business processes that rely on point-to-point integrations.
• Eliminates duplicate, redundant data feeds provided by third-parties.
• Allows cleanse all data across the enterprise thanks to integration of data from disparate applications into a single system.
• Centralizes data across the enterprise which helps reduce the amount of redundant data storages and systems. Thus cuts the cost of needless licenses, support and hardware.
• Configurable off-the-shelf MDM platform replaces antiquated custom applications saving significant development and maintenance costs associated with band-aiding custom solutions.
• Eliminates reporting errors, expedite audits and improve compliance by
  a) managing a single version of the truth along with a history of all changes, and
  b) delivering this information to any reporting, business intelligence or data warehouse.

However, one should not be overoptimistic about MDM and set his/her hopes on immediate results. There is a significant amount of work to be done to make MDM implementation successful. But when the work is done, and it is done right, MDM looks pretty beneficial. For apart from cost-cutting, it’s able to improve the ability to share, consolidate, and analyze business information quickly, both globally and regionally. And it makes it possible to rapidly assemble new, composite applications (software that combines the elements of a business activity in a coordinated application and user interface) out of accurate master information and reusable business processes.

Historically, data migration projects have had a tendency to fail; according to Bloor Research, as many as 60 percent do not succeed.  How to prevent failure when migrating your data? Here is a seven-step process:

1. Source system exploration. The first phase of a data migration project is to identify and explore the source data. The most appropriate route for identification is to group data, customer names, addresses and product descriptions based on the target model.
2. Data assessment. The next logical phase is assessment of the quality of this source data. If the new system fails due to data inconsistencies, incorrect or duplicate data, there is very limited value in migrating data to the target system. To assess the data, profiling the data is highly recommended.Through the use of data profiling, you can:
   •  Immediately identify whether the data will fit the business purpose.
   •  Accurately plan the integration strategy by identifying data anomalies up front.
   •  Successfully integrate the source data using an automated data quality process.
3. Migration design. In the migration design phase, the main tasks will be to define the technical architecture and design of the migration processes.  In addition, you will define the testing processes and how to transition them to the production system. You will also determine whether there will be a parallel run, a zero-downtime migration, or whether you will be expecting to complete the migration and simply “decommission” the old system.
4. Migration build. Be careful about using a “just enough” development approach with your data migration, simply because the migration will be executed only once and there will be limited reuse of the code. These assumptions explain why data migrations are prone to failure.
5. Execution. After comprehensive testing, the time comes to run the migration. In the majority of cases, the source systems are shut down while the migration executes. To minimize impact, this is likely to occur over a weekend or public holiday. In some cases, where the source applications are required to run 24/7, 365 days a year, a zero-downtime migration approach may be needed.
6. Transition. At some point after the data has been migrated, decide when to move to the new system, and where appropriate, retire the old system. During the execution phase, audit trails and logs will be created to ensure that all data has been correctly migrated and, when appropriate, that the correct synchronization has been achieved. Finally, after reviewing the audit trails and logs, you will be prepared to make the decision to transition users to the new system.
7. Production. As part of the design process, a system retirement policy will be created to address the old system. There will also be ongoing data quality enhancements. Remember that because not all source systems will be retired, data quality issues may be identified and will need rectifying in several of the source systems. You will also need to manage ongoing improvements and monitor the data quality of the new system.