It’s a sad fact, but many organizations realize the data quality of the data bases they work with is poor, only when it comes to database integration. Data quality issues are among the common reasons for data integration failure.

The reason for such a neglecting attitude to data quality lies in the fact that companies often don’t understand how much data quality impacts business processes. Thus, each data source or database a company uses may have its own rules and standards for data quality. The issues, however, evolve as soon as the database integration started in order to get a unified look at, for example, company’s customers’ data.

Those issues may come out of the difference of data fields, for example, or data formats, so the same contact may be represented differently in different databases. Thus when it comes to database integration, it can’t be performed right due to those differences, which may lead to data duplication, and many more data quality issues. In fact, in the result of integrating several databases of poor quality, a company gets one big database of poor quality. This means that database integration was in vain, as it failed to achieve its main goal of providing the company with a general view of business data, while the integration expenses were significant.

Unfortunately, data quality technology does not always allow organizations to fix poor data. And, it’s much wiser to implement company-wide standards for data quality to prevent the appearance of data quality issues associated with integration of data from heterogeneous sources, then to perform data cleansing and other data quality procedures afterward.

Data Integration Is Not About Tools, It’s About Strategy

Today, organizations face increasing data integration challenges. The amounts of data increase progressively, demanding for new levels of data protection, and making data migration even more complex.

At the same time, business demands access to a real-time information, and quality data to make right business decisions. There are plenty of technologies that are able to address data integration challenges, though some of them get old-fashioned, some continue to mature, etc. Thus according to Forrester Research, MDM and data quality services continue to mature, while ETL (extract, transform, and load) and data replication “have reached the Equilibrium phase,” and some technologies are moving to a decline.

But that doesn’t mean that a technology or tools which are in the top of the list today, may become a successful solution for data integration challenges. Lots of organizations regard data integration as mostly a technological process, not taking into account how it impacts organization’s long-term plans and the success (or failure) of business. However, successful data integration is mostly about strategy. And when a strategy is defined, the choice of tools get’s much easier, and there’s no risk that a chosen technology won’t cope with the task.

Data migration is a complex process, and it differs greatly from other IT projects. Good sound approaches should be included when planning data migration strategy. Success of data migration depends on many things and each detail is important. But there are certain conditions that affect the complexity of data migration process, and thus demand even more attention.

One of such challenging conditions is moving from a current vendor to a new one, which may mean another type of applications and systems, different data formats, etc. This may, probably, demand the use of data migration tools different from those, a company had utilized.

Another condition that complicates data migration is moving from physical environment to virtualized one. The same challenge is observed if we speak about environments in the Cloud. Though the end result of the shift is going to be great, a much greater effort needed to overcome difficulties, connected with the process (lack of interoperability between different cloud and physical platforms, security provisions, access to the data, etc.)

So, as new possibilities for storing, accessing, and working with data appear, promising significant decrease in expenses and resources, organizations are to adopt them. However, one should bear in mind that there are certain things and conditions that may complicate the process of data migration, and provide for them.

A CRM is a great solution to effectively manage company’s customer data. However, to make the management effective, get the most out of CRM system, and avoid CRM failure, special attention should by paid to what data is migrated, and how it is migrated.

Data migration from various sources to a CRM is a challenge, so it should be approached with care and strategy. Migrating the data just as is leads to a CRM failure, as you simply take siloed data and move it to another system (the data still remains siloed). So, before data migration takes place, there is a work that should be done to ensure the quality of the data that is targeted to a CRM.

Data mapping is a challenge. To migrate data, one should map the source and the target. That’s clear. The problem is, that in different sources data fields possessing equal information, may be named differently (for example, “Username” and “Account”), or vice versa, equally named fields may stand for different values (“Name” may stand for a first name solely, or the full name, including first name and surname). That’s why it’s very important to map fields in a data source and with appropriate fields in a CRM application.

It’s highly probable that before data migration started, data will need to be transformed, in order to appropriate for a new CRM system. Surely, it should be cleaned from duplications, incomplete records, outdated records, etc.

There is a variety of ETL tools available on the market, starting with solutions with minimum functionality and ending with tools that help solve complex tasks. The choice is also between proprietary offerings and open source ETL tools, Web based and desktop.

When choosing an ETL tool for a particular company, a lot of things should be taken into consideration, including currently used data management processes, technologies utilized, IT staff available, etc.

Thus, to evaluate ETL tools and make a decision in favor of certain offering, a set of questions should be answered, such as the operating systems supported by an ETL tool, the volume of data the tool is able to handle in a given period of time, data sources and data formats the ETL tool supports, etc. There is also significant to find out on what conditions maintenance and support are provided (paid, free of charge, etc.)

It’s also important to analyze company’s requirements to data integration, and compare them with the functionality that different ETL tools provide before making a decision and purchase a tool. Thus, a company may avoid paying for the functionality they are not going to use.

“The data, and integration strategies around the data, is something that most figure is there, will be there, and requires very little thinking and planning.” – This is what I’ve read today at ebizQ.

This again supports the idea that data, though being “the biggest companies’ asset”, is still being often neglected. And this results in data integration and quality issues, providing inconsistent data and ruining the entire idea of data integration as a way to provide a clear view on enterprise data. The data that is important for business decisions.

In fact, very often when it comes to designing and developing the architecture, all the attention is focused on technical side of the process. Thus data integration strategies become an afterthought making it difficult to meet business requirements. So, the message is that provisions for data integration should be made at the level of the development of enterprise systems architecture.

There’ve been lots of talks about the need of a union between data integration and data quality solutions, as no integration initiative can be called successful without the quality of the data provided.

Though data quality functionality becomes a required feature in data integration efforts, it seems to remain a challenge. According to Gartner, only 25% of organizations will proactively and comprehensively include data quality processes and competencies in their data integration work in 2010. This might mean that a large number of enterprises will have to struggle with incomplete, inconsistent and inaccurate data, as their main focus is on the mechanics of data integration process with no connection to data quality.

However technology providers apply more efforts on making data integration and data quality interoperable, providing the possibility to combine these data management processes into one. Thus the step is made toward the consolidation of technology offerings, instead of using distinct data integration and data quality solutions. The pros of this consolidation is quite obvious – a consistent and complete view of the business data, and as a result, the ability to change business processes and models faster to gain the market.

Admitting that Cloud, has become a real opportunity to simplify the work with applications, cut hardware costs, management costs, etc., even large enterprises start their ways to the Cloud.

The challenge is that it’s not enough to migrate a portion of enterprise application to the cloud (with migration being a challenge itself), the next step is to provide integration of cloud applications and in-house apps and systems. Cloud will not bring any value, until data integration is provided; otherwise, an enterprise receives a data silo not synchronized with on-premise systems.

And here come data integration complexities.

Interoperability is a pain of Cloud platforms. They emerge fast, and interoperability is not taken into account. This complicated data integration initiative for enterprises, as the choice of the tools will depend on the platform (so if more than one platform chosen, there’s problem).

Another thing is that cloud data integration requires rather new approaches and data integration mechanisms. Thus there will be a demand for new data integration tools, as well.

Moreover, different cloud platforms have different levels of security. As data integration between the Cloud and on-premise applications means moving sensitive data across them (and the amount of data will be increasing), there should be developed security standards to protect that data in transit.

According to one of the latest researches by Aberdeen Group, issues with customer data may greatly affect organizations’ sales and marketing efforts. Besides, lack of trusted customer data and inability to target customers through customer data proved to be the top two of those issues, according to the respondents’ opinion.

This reveals the fact, that there’s still a lack of customer data integration and data quality at many organizations.

This seems strange to me, especially taking into account that experts have been talking a lot about the importance of customer data integration across the enterprise to give all the responsible a better view on who are their customers, what do they need and buy.

Even in a rather small company there could be several sources containing customer data, and not obviously those sources contain similar records, so customer data integration is the means to piece together the data puzzle.  Probably, mergers the number of which was significant in 2009 prevented numerous organizations from fulfilling their customer data integration initiatives.

Well, the fact is that more than 60 percent of Aberdeen’s survey respondents named customer data integration and other customer data initiatives among their top priorities for 2010.

According to multiple predictions and publications, 2010 is going to become quite an interesting year for open source data integration.

Here, as you may remember, Gartner has proclaimed open source solutions “good enough” for data integration (extract, transform, and load, to be exact), and a bit later has mentioned (at last!) open source data integration and BI vendors in its Magic Quadrant, thus admitting that open source solutions can be mature enough to meet their functionality requirements.

Though sometimes there are still talks about the need to have skilled developers at hand, for the sake of support and maintenance, it seems that open source data integration tools move closer to becoming a mainstream, and not just a cheap alternative (with limited possibilities) to proprietary data integration solutions.

Proprietary BI and Data integration vendors seem to admit this fact, as, according to Gartner, some of them have introduced free “starter editions” of their solutions.

All this brings us hopes that times, when open source data integration tools were regarded just offerings for small and mid-sized businesses, are passing, and open source offerings will gain the right to be deployed in large enterprises alongside commercial proprietary BI solutions.