Admitting that Cloud, apart from being a buzzword, has become a real opportunity to simplify the work with applications, cut hardware costs, management costs, etc., even large enterprises start their ways to the Cloud.

The challenge is that it’s not enough to migrate a portion of enterprise application to the cloud (with migration being a challenge itself), the next step is to provide integration of cloud applications and in-house apps and systems. Cloud will not bring any value, until data integration is provided; otherwise, an enterprise receives a data silo not synchronized with on-premise systems.

And here come data integration complexities.

Interoperability is a pain of Cloud platforms. They emerge fast, and interoperability is not taken into account. This complicated data integration initiative for enterprises, as the choice of the tools will depend on the platform (so if more than one platform chosen, there’s problem).

Another thing is that cloud data integration requires rather new approaches and data integration mechanisms. Thus there will be a demand for new data integration tools, as well.

Moreover, different cloud platforms have different levels of security. As data integration between the Cloud and on-premise applications means moving sensitive data across them (and the amount of data will be increasing), there should be developed security standards to protect that data in transit.

According to one of the latest researches by Aberdeen Group, issues with customer data may greatly affect organizations’ sales and marketing efforts. Besides, lack of trusted customer data and inability to target customers through customer data proved to be the top two of those issues, according to the respondents’ opinion.

This reveals the fact, that there’s still a lack of customer data integration and data quality at many organizations.

This seems strange to me, especially taking into account that experts have been talking a lot about the importance of customer data integration across the enterprise to give all the responsible a better view on who are their customers, what do they need and buy.

Even in a rather small company there could be several sources containing customer data, and not obviously those sources contain similar records, so customer data integration is the means to piece together the data puzzle.  Probably, mergers the number of which was significant in 2009 prevented numerous organizations from fulfilling their customer data integration initiatives.

Well, the fact is that more than 60 percent of Aberdeen’s survey respondents named customer data integration and other customer data initiatives among their top priorities for 2010.

When migrating data to the cloud, ensuring data quality is essential.  Data can’t be taken to the cloud as is, so before starting data migration, provisions should be made for data quality.

It’s wrong to start data migration, until data is checked to be accurate, complete, duplications are found and cleared up, etc. Otherwise, data issues will be taken to the cloud, which will make it inconvenient to work with the data.

One more thing to be taken into account before beginning data migration process is a provider’s possibility to provide fresh, real-time data, and give constant access to the data.

Normally, companies have best practices for data quality. Cloud providers also have tools for data management, so when those tools and company’s best practices are united, it makes data management more flexible, and thus a company will have possibility to control data quality when data is migrated to the cloud.

According to multiple predictions and publications, 2010 is going to become quite an interesting year for open source data integration.

Here, as you may remember, Gartner has proclaimed open source solutions “good enough” for data integration (extract, transform, and load, to be exact), and a bit later has mentioned (at last!) open source data integration and BI vendors in its Magic Quadrant, thus admitting that open source solutions can be mature enough to meet their functionality requirements.

Though sometimes there are still talks about the need to have skilled developers at hand, for the sake of support and maintenance, it seems that open source data integration tools move closer to becoming a mainstream, and not just a cheap alternative (with limited possibilities) to proprietary data integration solutions.

Proprietary BI and Data integration vendors seem to admit this fact, as, according to Gartner, some of them have introduced free “starter editions” of their solutions.

All this brings us hopes that times, when open source data integration tools were regarded just offerings for small and mid-sized businesses, are passing, and open source offerings will gain the right to be deployed in large enterprises alongside commercial proprietary BI solutions.

The idea of cloud computing has become really popular; the increase of the cloud adoption in 2010 has been predicted by data integration experts. There’s no surprise, as cloud computing is positioned as a way of simplifying technology, so that wider audience could adopt it.

The need to synchronize data from applications that an enterprise has in the cloud with on-premise apps is bringing us to cloud integration (or data integration with the cloud).

Today there are different approaches to data integration with the cloud available, as well as different solutions to perform cloud integration. This variety is mostly comes out of the immaturity of the market, and, I suppose, with the development of the cloud, there will be defined right approaches to data integration. Among the most common approaches is customizing the same data integration tool that an enterprise uses to integrate in-house applications, so it could be used for the cloud applications, as well; moving a data integration tool to the cloud, which helps avoid expenses connected to hardware installation. One more way is on-demand data integration.

So, there is a variety of choices, and to make a decision on what approach to utilize for cloud data integration a company should understand their needs, as well as consider available budget and resources, etc.

Well, let’s have some words on customer data integration (CDI). CDI has been one of the main concerns for different organization. Making and keeping customer data clean and convenient to work with, and getting more value from CRM systems has been in the top five of data integration challenges.

As it was mentioned in one of researches by Forrester, many organizations have changed their view on customer data integration and management (and, generally, on MDM), and started looking at it as a multiyear investment, including several phases. Though, again according to Forrester, the right approach to customer data management, and customer data integration as its significant part, still keeps being elusive.

The problem is that many enterprise CRM applications often give quite fragmented view on enterprise’s customer data, due to poor customer data integration. One more issue, the quality of customer data is often poor, so this is one of the reasons of failed customer data integration.

So, as predicts Forrester, the question of customer data integration is not solved yet, and there is a lot to be done in 2010.

Okay, the cloud seems to become a real trend, so now for many enterprises there’s no question whether to adopt the cloud, but how to do it better.  Cloud’s major idea, that attracts many organizations, is the possibility to move their applications out of internal data centers into the cloud, and let cloud providers take care about maintaining the applications. However (and it’s been mentioned not once), application migration and data migration to external clouds are not simple processes.

Not to become overenthusiastic and have a clear view on moving to the cloud, it’s good to know the challenges and questions that data migration raises.

Clouds possess different architectures, and thus require technical personnel to upgrade their skills to fit cloud requirements in terms of implementation and operation. This is the most likely-to-face data migration challenge when moving applications to the cloud.

Data security is also among cloud data migration challenges. In fact, moving data into some third-party provider’s servers, you’ll know about it that it is somewhere in the cloud. This somewhere may be anywhere; there were lots of talks this year about clouds lacking transparency. However, this doesn’t mean that data migration into the cloud is a bad idea. This only means, that what applications (and what data) to move to the cloud should be carefully defined.

As for lowering cost, yes, moving to the cloud is one of the solutions to cut company’s expenses.  But that won’t come immediately. Initially, data migration will demand investment, for either data migration tool will be necessary to perform on-premise-cloud data migration, or the need to pay for data migration service. What matters, is that there won’t be free data migration anywhere. Depending on what tool you will choose, the cost may be higher or lower.  Yes, I’m talking about open source solutions that are available today as widely, as proprietary tools. However, when making a decision about what to choose, you shouldn’t be driven by merely cost cutting ideas, but follow your business goals, and choose the tool providing best opportunities to achieve those goals.

So, moving to the cloud has become a trend, and it’s fair, for the benefits provided of the cloud are real. However, though it seems simple just to move enterprise applications and data to the cloud, data migration must be properly planed and possible challenges analyzed before making the first move.

First of all, are they reluctant? According to an article at ebizQ I’ve stumbled upon, large enterprises “typically don’t deploy latest versions of anything, anytime soon.” This is more than fair when we speak about cloud computing. Small and mid-sized companies adopt cloud services much faster, notwithstanding the fact that cloud adopters may face security and data integration issues.

So, why data integration with the cloud is considered to be a complex problem?  There are different cloud platforms available today. Consequently, each cloud platform has its own API, having its own functionality and quality of services, which differ greatly from other cloud platforms’ APIs. So, here’s one of the reasons of the complexity of data integration between enterprise and cloud sources: additional efforts will inevitably be needed to provide for this difference.

Getting back to SMBs and large enterprises, SMBs are more flexible in adoption of evolving technologies. That’s because most SMBs are do not possess, or simply can’t afford IT departments responsible for data integration and other company’s IT needs. So cloud services offer real way out for such companies. As for data integration, small companies have relatively small amounts of data to be integrated between on-premise applications and applications residing in the cloud (that’s surely, compared to large enterprises).

Large enterprises have different requirements to data integration: their expectations and operational rhythm differ greatly from those of SMBs and mid-sized organizations. So there’s no surprise large enterprises are not hurrying into the cloud, waiting until all issues, including data integration, compliance and security, to be solved.

Data migration being one of the processes that are important for business, is a process implying several risks, as well. The issues that may be faced during data migration include security leaks, overrun budgets, project’s viability issues, etc.

To mitigate those risks effectively, risk-management strategy is necessary to be created on a pre-migration stage of the project. This strategy includes assessment of risks and challenges waiting ahead.
The migration plan should be created, including each phase of data migration. However, blindly following the plan would not be the right way of doing data migration. The plan should be revised and adapted to the issues discovered at each stage of the migration process.

Security issues are the most frequent reason for data loses and leaks. Very often security permissions are left behind, and important company’s data may be lost, corrupted or even stolen. Thus, data migration strategy should contain provisions for security issues.

Block-level migration is considered to be more risk-proof than file-level migration. While security settings in a file-based migration may be configured, this is possible when both source and target of data migration are situated in the same authorization domain.

Thus, there should be clearly defined what data is to be migrated, how the data will be copied, solely files, or file attributes and associated information, as well, etc. All this needs to be defined before data migration is started, and if needed, extra permissions and access levels should be created.

Surely, everything about what data is to be migrated, and the way it will be migrated should be mentioned in the documentation and discussed in details with data migration provider.

When deciding to start a data integration process, many companies consider using ETL tools instead of hand-coding. Such a decision is justified by the fact – and many data integration experts agree with it – that hand coding is error prone, takes time and additional resources, etc. However, it’s also wrong to assume that an ETL tool will help to finish data integration project sooner, or will result in some substantial cost savings, according to a TDWI.

Their point is that though ETL tools definitely accelerate the process of data integration at some level, one should not leave aside time that is to be spent on ETL tools evaluation, selection, and implementation.

Another deception is about cost savings. The acquisition cost of ETL tools is quite sufficient, and the annual support cost is often overlooked when a decision is being made on selection and implementation of an ETL tool. Thus, companies have a bit wrong idea about the amount of savings they might have.

The misconceptions described above, are a source of inappropriate expectations, and as a result, wrong assessment of data integration initiative expenses, and at worst, failed data integration initiative.

I think, the situation’s a bit different, when we speak about open source ETL tools. First, there’s no such thing as annual support cost. Huge developer and user communities make it possible to receive support from other users, without paying for it. Then, license costs of open source ETL solutions are really low, which allows to redirect the released budget where there would be a demand for additional finance. So, here I see a real possibility to reduce the cost of data integration with the help of ETL tools.

What I agree with, is that selection process will take time, as well as deployment (including user training), though open source solutions are typically easier to deploy, compared to proprietary ETL tools. Companies should take time for proper evaluation of ETL tools, either open source or proprietary; and I do agree that the decision should be taken based on whether an ETL tool fits this peculiar company’s business needs best and is capable to provide a company with help in achieving their goals.