The extensive world of enterprise data is quite tricky; however, it’s very important to clearly distinguish all the solutions used when dealing with data.

As customers are very important for any company, customer data is among the greatest company’s assets. In fact, customer data integration (CDI) solutions are to deliver the fullest information about customers.

However, as CDI is a software solution, and, in fact, is a clearinghouse for data synchronization and deployment, it is inevitably compared to data warehouses. There are several reasons of this confusion:

• the aim of both solutions is to accommodate clean, meaningful information to the enterprise
• both solutions are undoubtedly beneficial for business
• both demand a solid partnership of business and IT

This confusion is risky, for these solutions differ in term of their positioning as well as in term of their usage. According to Jill Dyché one of the most acknowledged BI experts:

Data warehouses are designed and built to support business intelligence, and are meant for use by business people. Best practice data warehouses are those that have been planned around a set of business requirements that inform a series of applications—we call this the BI Portfolio—that are deployed incrementally to the business over time.

CDI, however, is purpose-built for operational data integration. The CDI hub is the ultimate home of customer master data that has been matched, reconciled, and certified, and is available to a series of business applications and systems (not end users). Unlike the data warehouse, which usually stores historical detail, summarized data, and time-variant information, the CDI hub stores or points to certified master data about a customer.

Data migration is an inevitable evil for any enterprise. As business requirements change and demand for new applications, or there is a need of removing duplications and inefficiencies, business-critical data migration can’t be avoided. Data migration initiatives, unfortunately, still have a high rate of failures due to the complexity of the process. According to CRN Australia’s article I’ve found lately, there are four common data migration rulles which could help to drive the initiative to success.

The first rule says that data migration is primarily a business issue. Technical side is important, but only as a means to fulfill the process. IT people don’t always have the power or the necessary knowledge to deliver what is required by the business. It is business who is to make the decision on:

• What is the purpose of migrating data?
• What data should be migrated?
• When should the data be migrated?

The second rule claims that business goals define the solution and approach selected for data migration. This rule supports the first one. It is important to bear in mind that the best technical solution is not always the best business solution. IT’s responsibility is to make sure the chosen migration technology is able to support possible changes in priority and direction without restarting every time. IT also should provide for the issues that may occur during the migration.

The third rule is about setting the level of acceptance of data quality level. Many data migrations have been scuppered by overestimating or not understanding the quality of the source data. While enhancing data quality is a worthy goal, it is really important not to go off on a data perfection crusade. It is a trap that drives many, many projects to inflating both the cost and the time to deliver the project. To avoid this trap, data owners and users need to determine the level of quality they require at the start of the project, in order that the technologists have an appropriate goal.

The fourth rule is about measuring data quality in order to determine the level of quality business users require. Those measures should make sense to business users and not just to technologists. It makes possible to rightly measure deliverables, perform gap analyses, and monitor and improve ongoing data quality. It also ensures that the efforts are concentrated on where business users see value and can quantify the benefits.

Open source data warehouses possess the same options as any other types of open source software: the same model of licensing, community development processes, and same degree of openness. They may be offered as free downloads, or for a nominal flat fee, as fully supported systems. Or there may be no limit to the number of licenses and implementations a company may make with the software.

Acording to BeyeNetwork article, the benefits of the open source data warehouses are following:

• Up front and maintenance costs are less than those of proprietary software. Besides, there is a possibility to customize the products companies use to improve their operations, for the original source code is open and may be downloaded.
• Skill sets that are widely available in the market are employed.  As a result, an organization with existing database or data warehouse expertise will not have to look further when a new open source data warehouse project is put into place.
• Improved standardization. Transparent and community supported open source code considers important standards to be consistently supported across all versions and implementations. Something that proprietary formats cannot and will not offer.
• Flexibility which enables enterprises to expand the solutions to an unlimited number of users, with no per-user or per-processor charges of proprietary software packages.
• Community effect. Open source solutions leverage communities of developers and innovators to advance development. New code and features are contributed back to the community, constantly increasing the range of new options available to end users.  Moreover, companies may address the community in order to fix any bugs or security flaws, which takes, normally, only days, instead of waiting weeks and months for the next security patch or service pack from a vendor.
• Incremental implementation.  There is no need to a mega project at once. Projects can start small and build upon the success of implementations. This dumps the tendency to “overpromise,” which is often a necessary evil for acquiring optimal levels of funding for data warehouse projects.

There’s been quite a period of time since open source data warehouses evolved and gained popularity. However, an open source data warehouse is still regarded as a solution for small or mid-sized companies lacking enough budgets for solid proprietary solutions. Bigger companies may also use open source solutions as complimentary to their proprietary data warehouses.
Getting the most out of an open source data warehouse implementation is possible. There are some ways below provided by Claudia Imhoff:

• Open source data warehouses complementing already existing proprietary enterprise solutions may help quickly address the new company’s needs. Proprietary solutions being more strategic are not so fast to react to those changes.
• Normally, it’s the analysts who work with data warehouses; they are familiar with building massive queries and other technical stuff. But in some cases, there are end users who don’t have special technical knowledge, and need as much ease of use as possible.
• Open source data warehouses should be compatible with related open source environments.
• While open source data warehouses may seem cheaper than proprietary solutions at first, additional costs, such as transition and training costs, should be taken into account.

CRM has become extremely popular in the recent years. Nowadays, many companies have accepted the importance of CRM and have made the decision to implement CRM initiatives enterprise-wide. However, a great deal of CRM strategies fails. In fact, those failures may be caused by different factors. But here are the most typical reasons of CRM projects failures:

• Believing that CRM starts and ends with software. However, a CRM project means the right people executing the right processes, using the best possible tools at their disposal.

• Under committing a CRM initiative – which means failing to invest enough time and attention to to develop a comprehensive sales process reengineering vision, settling instead for a series of minor, tactical changes.

• Considering CRM to be a part-time project. A part-time approach will not generate part-time results; it will generate no results at all. For CRM projects require full-time assignment of people for the duration of the project, active executive involvement, full-time commitment of personnel, accountability for results, and a process-driven budget.

• Having big expectations and small budgets. Trying to implement a CRM initiative as cheaper as possible is a mistake a company will regret very soon. A low-cost approach is extremely prone to failure. Starting a CRM initiative one should never focus primarily on price, but only on benefits.

• Picking wrong technology partners is a mistake that occurs when a company bases vendor choice only on product features. However a company must thing about their specific sales process functionality needs.

• Assuming that automating your sales process will be similar to automating manufacturing or finance. The degree of complexity in implementing a CRM system is significantly higher than the other two examples. Budgets should be provided for necessary upgrades and for ongoing system support.

Business intelligence is considered to be the helping hand for organizations wishing to do more with less. However, David Hatch, an Aberdeen Group analyst, has outlined four potential cost factors likely to arise in a BI initiative if an organization isn’t paying attention. According to Hatch, overall cost of ownership is not about the costs of purchasing the software, there are also indirect and hidden factors which affect the real BI costs. Very often the resources the company needs to acquire to properly implement, deploy, support, and maintain a BI solution are far more greater than it was assumed in the beginning.

Here are those factors, as Hatch describes them:

1. Year-after-year budget increases: The typical best-in-class company sees a drop in year-after-year BI budgetary costs. Average and laggard companies, however, can witness increases in BI expenses that range from 2 percent to 9 percent.
2. Cost per user: Best-in-class companies lower per-user costs by 4.3 percent whereas average performers and laggards often see increases ranging from 1 percent to 7 percent.
3. Time to complete projects: Best-in-class achievers complete BI projects, on average, within 14 days. Average performers take nearly three times as long (approximately 39 days) to complete a project, and the typical laggard company takes more than 12 times as long (177 days).
4. Modifications to BI software: Altering a BI program takes less than a day for best-in-class companies; three days for average performers; and up to eight days for laggard organizations.

Then, how a company can avoid additional costs of ownership and achieve higher returns from BI initiatives? Aberdeen suggests that investments in the following areas will maximize results:

• Data integration and cleansing: “Companies are finding it difficult to bring data together from multiple, disparate sources,” Hatch says. Investing in tools for data management can be of help in this regard. Best-in-class companies are twice as likely as their counterparts are to institute data integration and cleansing capabilities.
• End-user requirements: “You really have to stop and think about why…so many companies have deployed tools that so many aren’t able to use,” Hatch says. Companies must understand that end-users — especially nontechnical, non-data-guru types — may need different approaches. Hatch advises companies to focus on end-user needs before deploying a solution.
• Training: Top performers are 37 percent more likely to invest in extensive user training on BI solutions and 40 percent are more likely to have formed formal user committees to encourage adoption. Additionally, best-in-class companies are twice as likely as laggards and average performers are to sign up for vendor-provided services.
• Operational BI: Successful users of BI use the technology on an everyday basis rather than merely getting a summarized spreadsheet version of performance and high-level trends. Hatch says that operational BI seems to be gaining traction as companies look to make comparisons over shorter time spans rather than just examine large-scale trends.

Customer data integration (CDI) projects demand transformation of people, process and technology. If just one area fails the entire project will be at risk.

Here are common pitfalls that may turn a CDI initiative into failure, according to SearchDataManagement.com:

• Data model misfortunes
  Vendors usually promote those data models were designed to support their applications. They can’t offer flexibility around the data model. However, a predetermined data model, designed by an application vendor, can be a big reason for CDI failure.  Companies should look carefully at various data model approaches and options and choose those answering their needs.
• Overarching architecture and technology issues
  While opinions again vary on the right technical and architectural approach, it’s clear that there is no “one size fits all” approach to CDI. A company’s existing systems, potential longer-term changes and specific business requirements all play a part in determining the correct technological approach for a given project. Issues of CDI system architecture, performance and scalability are all very important to consider up front, and flexibility is critical.
• No plan (or budget) for long-term maintenance and extensibility
  CDI is not a one-shot deal. It’s an ongoing project. Early adopters of CDI, many of whom built their own in-house systems, are now looking toward commercial products. They have found out about the long-term costs and requirements of maintaining custom systems.
• Lack of user adoption
  Education and training are extremely important parts of CDI implementation and must be a primary part of a CDI plan, not an afterthought. Many companies using CDI successfully recommended developing a corporate-wide marketing plan for a CDI project and associated data governance and data stewardship initiatives.
• Not addressing data quality, governance and stewardship issues
  CDI projects without an associated enterprise-wide focus on data stewardship and data governance will fail, many attendees said, because the source of the data problems might not be addressed. Worse yet, if users discover data problems in the context of a CDI implementation and start to distrust the data in the system, the whole CDI project can fail due to skepticism leading to lack of user adoption.
• Politics, pure and simple
  Underlying many CDI-challenge discussions was the issue of corporate politics. By definition, CDI projects can touch almost every department in an organization. Whether it’s the lack of enterprise-wide support, unrealistic timelines, inadequate budgets or data ownership issues, CDI project leaders must be social champions, as well as architectural experts. Data management can be a touchy issue, according to many attendees. Various divisions may feel that they own the data in their own systems and may be reticent to allow another system to access — much less change — what they consider to be their critical information.

At this stage several important questions should be asked and answered about specifications and requirements. By the end of this phase you should have a thorough specification of how the source and target objects will be mapped to pass the results to a developer for implementation in a data migration tool. You also should have a firm interface design, data quality specifications, the firm idea of what technology will be required in the production environment, and, finally, service level agreements.

Here are the details:

• Have you created a detailed mapping design specification?
  Note that there is no immediate progress into build following landscape analysis. It is far more cost-effective to map out the migration using specifications as opposed to coding which can prove expensive and more complex to re-design if issues are discovered.
• Have you created an interface design specification?
  A firm design is needed for any interface designs that are required to extract the data from your legacy systems or to load the data into the target systems. For example, some migrations require change data capture functionality so this needs to be designed and prototyped during this phase.
• Have you created a data quality management specification?
  This will define how you plan to manage the various data quality issues discovered during the landscape analysis phase. These may fall into certain categories such as:
  •     Ignore
  •     Cleanse in source
  •     Cleanse in staging process
  •     Cleanse in-flight using coding logic
  •     Cleanse on target
• Have you defined your production hardware requirements?
  The volumetrics and interface throughput performance should be known so you should be able to specify the appropriate equipment, RAID configurations, operating system etc.
• Have you agreed the service level agreements for the migration?
  At this phase it is advisable to agree with the business sponsors what your migration will deliver, by when and to what quality.
  Quality, cost and time are variables that need to be agreed upon prior to the build phase so ensure that your sponsors are aware of the design limitations of the migration and exactly what that will mean to the business services they plan to launch on the target platform.

Why enterprise-wide data quality initiative often turns into a total disappointment? According to Forrester’s report “A Truism For Trusted Data: Think Big, Start Small,” it happens because of managers’ ambitions to implement an enterprise-wide system for trusted data all at once.

However, experts from Forrester recommend thinking global, but starting small. In other words, they advice consider a bottom-up approach that defines quantitative and qualitative ROI for only those few select functional organizations that can best articulate and measure the business impact poor quality data has on processes.

Here is a short overview of the steps, Forrester recommends, to effectively implement data quality initiatives:

1. First of all, those responsible should start from defining what they mean by “data quality” and “trusted data”. As Jill Dyché once mentioned, it’s time to understand the following seldom-understood truth: That there are different levels of “acceptability” for data. And, according to her, the key is to understand company’s business requirements and then drill them down to data requirements. That will tell conclusively what good enough for the company really is.

Forrester defines “data quality”  as “data used by business stakeholders to support their processes, decisions, or regulatory requirements with no reservations as to the data’s relevance, accuracy, integrity, and other previously agreed upon definitions of quality.”

Forrester reminds that data quality must come directly from business stakeholders, for they are those who understand business requirements of the company, and thus may set the standards for company’s data quality.

2. Then, Forrester insists on building a business case that starts small.

“Scoping and prioritization based on the business processes within the organization that are most critically affected by poor data quality is the key to defining the business case that will get your trusted data initiative off the ground,” say experts.

3. As word of the value of the data quality project gets around, other organizations, such as those responsible for order management and fulfillment, may also want to implement data quality improvements within their environments.

“Eventually, the tide will turn and these business stakeholders will sign up and support an enterprise-class solution to solving their data quality problems, but for that to happen, value must be demonstrated,” Forrester concludes.

Actually, this phase is more a kind of a sequel of the previous one, so it would be better called Phase 2.2. You know, now I understand why so many enterprise data migration initiatives go wrong, you need to bear in mind soooo many things. The process demands constant interaction of all the people involved… Okay, let’s continue checking the data migration to-do list.

• Have you determined high-level volumetrics and created a high-level scoping report?
  Ensure that you fully assess the scope and volume of data to be migrated.Focus on pruning historical and redundant data. Create a final scoping report detailing what will be in scope for the migration and get the business to sign this off.
• Has the risk management process been shared with the team and have they updated the risk register?
  Create a simple online form where anyone can add risks during their analysis, you can also filter them out later but for now we need to gather as many as possible and see where any major issues are coming from.
• Have you created a data quality management process and impact report?
  If you’ve been following our online coaching calls you will know that without a robust data quality rules management process your project will almost certainly fail or experience delays. Understand the concept of data quality rules discovery, management and resolution so you deliver a migration that is fit for purpose. The data quality process is not a one-stop effort, it will continue throughout the project but at this phase we are concerned with discovering the impact of the data so decisions can be made that could affect project timescales, deliverables, budget, resourcing etc.
• Have you created and shared a first-cut system retirement strategy?
  Now is the time to begin warming up the business to the fact that their beloved systems will be decommissioned post-migration. Ensure that they are briefed on the aims of the project and start the process of discovering what is required to terminate the legacy systems. Better to approach this now than to leave it until later in the project when politics may prevent progress.
• Have you created conceptual/logical/physical and common models?
  These models are incredibly important for communicating and defining the structure of the legacy and target environments.
• Have you refined your project estimates?
  Most projects start with some vague notion of how long each phase will take. Use your landscape analysis phase to determine the likely timescales based on data quality, complexity, resources available, technology constraints and a host of other factors that will help you determine how to estimate the project timelines.