The idea of cloud computing has become really popular; the increase of the cloud adoption in 2010 has been predicted by data integration experts. There’s no surprise, as cloud computing is positioned as a way of simplifying technology, so that wider audience could adopt it.

The need to synchronize data from applications that an enterprise has in the cloud with on-premise apps is bringing us to cloud integration (or data integration with the cloud).

Today there are different approaches to data integration with the cloud available, as well as different solutions to perform cloud integration. This variety is mostly comes out of the immaturity of the market, and, I suppose, with the development of the cloud, there will be defined right approaches to data integration. Among the most common approaches is customizing the same data integration tool that an enterprise uses to integrate in-house applications, so it could be used for the cloud applications, as well; moving a data integration tool to the cloud, which helps avoid expenses connected to hardware installation. One more way is on-demand data integration.

So, there is a variety of choices, and to make a decision on what approach to utilize for cloud data integration a company should understand their needs, as well as consider available budget and resources, etc.

Well, let’s have some words on customer data integration (CDI). CDI has been one of the main concerns for different organization. Making and keeping customer data clean and convenient to work with, and getting more value from CRM systems has been in the top five of data integration challenges.

As it was mentioned in one of researches by Forrester, many organizations have changed their view on customer data integration and management (and, generally, on MDM), and started looking at it as a multiyear investment, including several phases. Though, again according to Forrester, the right approach to customer data management, and customer data integration as its significant part, still keeps being elusive.

The problem is that many enterprise CRM applications often give quite fragmented view on enterprise’s customer data, due to poor customer data integration. One more issue, the quality of customer data is often poor, so this is one of the reasons of failed customer data integration.

So, as predicts Forrester, the question of customer data integration is not solved yet, and there is a lot to be done in 2010.

Okay, the cloud seems to become a real trend, so now for many enterprises there’s no question whether to adopt the cloud, but how to do it better.  Cloud’s major idea, that attracts many organizations, is the possibility to move their applications out of internal data centers into the cloud, and let cloud providers take care about maintaining the applications. However (and it’s been mentioned not once), application migration and data migration to external clouds are not simple processes.

Not to become overenthusiastic and have a clear view on moving to the cloud, it’s good to know the challenges and questions that data migration raises.

Clouds possess different architectures, and thus require technical personnel to upgrade their skills to fit cloud requirements in terms of implementation and operation. This is the most likely-to-face data migration challenge when moving applications to the cloud.

Data security is also among cloud data migration challenges. In fact, moving data into some third-party provider’s servers, you’ll know about it that it is somewhere in the cloud. This somewhere may be anywhere; there were lots of talks this year about clouds lacking transparency. However, this doesn’t mean that data migration into the cloud is a bad idea. This only means, that what applications (and what data) to move to the cloud should be carefully defined.

As for lowering cost, yes, moving to the cloud is one of the solutions to cut company’s expenses.  But that won’t come immediately. Initially, data migration will demand investment, for either data migration tool will be necessary to perform on-premise-cloud data migration, or the need to pay for data migration service. What matters, is that there won’t be free data migration anywhere. Depending on what tool you will choose, the cost may be higher or lower.  Yes, I’m talking about open source solutions that are available today as widely, as proprietary tools. However, when making a decision about what to choose, you shouldn’t be driven by merely cost cutting ideas, but follow your business goals, and choose the tool providing best opportunities to achieve those goals.

So, moving to the cloud has become a trend, and it’s fair, for the benefits provided of the cloud are real. However, though it seems simple just to move enterprise applications and data to the cloud, data migration must be properly planed and possible challenges analyzed before making the first move.

First of all, are they reluctant? According to an article at ebizQ I’ve stumbled upon, large enterprises “typically don’t deploy latest versions of anything, anytime soon.” This is more than fair when we speak about cloud computing. Small and mid-sized companies adopt cloud services much faster, notwithstanding the fact that cloud adopters may face security and data integration issues.

So, why data integration with the cloud is considered to be a complex problem?  There are different cloud platforms available today. Consequently, each cloud platform has its own API, having its own functionality and quality of services, which differ greatly from other cloud platforms’ APIs. So, here’s one of the reasons of the complexity of data integration between enterprise and cloud sources: additional efforts will inevitably be needed to provide for this difference.

Getting back to SMBs and large enterprises, SMBs are more flexible in adoption of evolving technologies. That’s because most SMBs are do not possess, or simply can’t afford IT departments responsible for data integration and other company’s IT needs. So cloud services offer real way out for such companies. As for data integration, small companies have relatively small amounts of data to be integrated between on-premise applications and applications residing in the cloud (that’s surely, compared to large enterprises).

Large enterprises have different requirements to data integration: their expectations and operational rhythm differ greatly from those of SMBs and mid-sized organizations. So there’s no surprise large enterprises are not hurrying into the cloud, waiting until all issues, including data integration, compliance and security, to be solved.

Data migration being one of the processes that are important for business, is a process implying several risks, as well. The issues that may be faced during data migration include security leaks, overrun budgets, project’s viability issues, etc.

To mitigate those risks effectively, risk-management strategy is necessary to be created on a pre-migration stage of the project. This strategy includes assessment of risks and challenges waiting ahead.
The migration plan should be created, including each phase of data migration. However, blindly following the plan would not be the right way of doing data migration. The plan should be revised and adapted to the issues discovered at each stage of the migration process.

Security issues are the most frequent reason for data loses and leaks. Very often security permissions are left behind, and important company’s data may be lost, corrupted or even stolen. Thus, data migration strategy should contain provisions for security issues.

Block-level migration is considered to be more risk-proof than file-level migration. While security settings in a file-based migration may be configured, this is possible when both source and target of data migration are situated in the same authorization domain.

Thus, there should be clearly defined what data is to be migrated, how the data will be copied, solely files, or file attributes and associated information, as well, etc. All this needs to be defined before data migration is started, and if needed, extra permissions and access levels should be created.

Surely, everything about what data is to be migrated, and the way it will be migrated should be mentioned in the documentation and discussed in details with data migration provider.

When deciding to start a data integration process, many companies consider using ETL tools instead of hand-coding. Such a decision is justified by the fact – and many data integration experts agree with it – that hand coding is error prone, takes time and additional resources, etc. However, it’s also wrong to assume that an ETL tool will help to finish data integration project sooner, or will result in some substantial cost savings, according to a TDWI.

Their point is that though ETL tools definitely accelerate the process of data integration at some level, one should not leave aside time that is to be spent on ETL tools evaluation, selection, and implementation.

Another deception is about cost savings. The acquisition cost of ETL tools is quite sufficient, and the annual support cost is often overlooked when a decision is being made on selection and implementation of an ETL tool. Thus, companies have a bit wrong idea about the amount of savings they might have.

The misconceptions described above, are a source of inappropriate expectations, and as a result, wrong assessment of data integration initiative expenses, and at worst, failed data integration initiative.

I think, the situation’s a bit different, when we speak about open source ETL tools. First, there’s no such thing as annual support cost. Huge developer and user communities make it possible to receive support from other users, without paying for it. Then, license costs of open source ETL solutions are really low, which allows to redirect the released budget where there would be a demand for additional finance. So, here I see a real possibility to reduce the cost of data integration with the help of ETL tools.

What I agree with, is that selection process will take time, as well as deployment (including user training), though open source solutions are typically easier to deploy, compared to proprietary ETL tools. Companies should take time for proper evaluation of ETL tools, either open source or proprietary; and I do agree that the decision should be taken based on whether an ETL tool fits this peculiar company’s business needs best and is capable to provide a company with help in achieving their goals.

The year’s end is closer, and traditionally, it’s time for next year predictions. And it seems that it’s a good time for cloud computing, for, according to multiple predictions, it will continue gaining popularity in 2010, and as a result, the rate of cloud adoption will increase. And that’s no surprise, for clouds turn beneficial for both IT and business users. What’s more, data integration growth will be driven mostly by cloud computing (that’s according to David Linthicum, a data integration specialist).

You know, there’ve been lots of talks this year about different issues of cloud computing, including security and data integration issues. However, I think most of those issues are not so hard to avoid. Let’s take data integration.

Seriously, many enterprises do not create any data integration strategy, and get really surprised when it comes to data integration and synchronization of cloud and on-premise systems. However, to avoid data loses and all the mess that will inevitably come together with the absence of data integration strategy, some efforts should be done before (and here the word “before” is essential) moving any enterprise systems to the cloud. This move should be prepared: data integration requirements for every system should be defined, architecture requirements specified, identified data sources that would need to be synchronized, etc.

So, I’m about to say, that the “issue” is not data integration itself, it is the absence of reasonable thinking, and so-called “cloud rush” (actually, rush is a phenomenon peculiar for any trendy technology adoption). Provide for data integration between both on-premise and systems moved to the cloud; don’t move to the cloud all at once, make it a gradual shift instead, ensuring successful data integration between newly shifted systems and the rest of enterprise – these are the recommendations to avoid data integration issues with the cloud.

In one of their latest press releases, Gartner made some interesting predictions regarding open source ETL and other kinds of BI tools. Though it’s early to put open source data integration platforms on the same level with complex proprietary solutions, the deployment of open source tools shows solid growth.

Gartner has pointed out the fact that open source ETL and BI tools are more frequently used by mid-sized businesses, governmental and public sectors. And many ISVs use open source BI solutions as the additional functionality to their own applications.

What is even more interesting: large vendors, proprietary commercial tools providers seriously care about finding the ways to address the challenges from open source data integration and BI offerings, as the latter become their lower-cost competitors.

Assuming this, I think that open source data integration, ETL and other kinds of business intelligence solutions have earned confidence from the business side, and are expected to become even more widely used in the nearest future.

I suppose this posting to be a kind of a continuation of the previous one. Here I’m again about ETL and data integration solutions selection, but I’d like to concentrate on open source ETL. I won’t make any discovery if, again, repeat that today’s open source solutions are good enough for ETL operations, and  data integration and BI experts are expecting them to develop into solutions for master data management.

But today, open source ETL provides alternative to proprietary solutions which are usually costly and supposed to be used for more complex data integration processes, apart from mere ETL. However, for mid-sized and small businesses that, as a rule, have smaller budgets and smaller open source ETL solutions are a means to address their data integration needs.

But I was about business requirements, or rather how open source ETL tools may address company’s business requirements for data. Here I see several ways:

First, if a company by chance has a couple of their own developers, they could make necessary customization to company’s ETL, thanks to the availability of the code.

Then, as a rule, open source solutions are supported by developers’ communities, some of which are really powerful. So, the community behind the open source ETL that a company uses may help with needed functionality or customization of existing ones to meet company’s business requirements.

And don’t forget about the vendor itself. A company may address directly to the vendor of their open source ETL and require additional functionalities that meet company’s peculiar needs.

And, as a rule, any of the actions described above will cost less and the result will take less time to deliver than in case with proprietary data integration tools.

Well, though the posting sounds so bright, there still may be issues with open source, such as vendors that stop supporting their solutions, etc. However with the communities behind, and thanks to the openness of the code, the chances to overcome those issues seem to me higher than in case with proprietary solutions.

In my previous posting, I touched upon the importance of defining business requirements before starting data integration initiative. Without those requirements, the process will comprise just blind gathering all kinds of data available at the enterprise with no clear purpose. In other words, data integration initiative may turn into some kind of monkey business. Okay, that’s clear.

Well, data integration tools selection, including ETL (extract, transform, and load) solutions, is a job that requires efforts but if done right, it’s worth it. What do I mean by this “done right”? The message is simple. When choosing an ETL tool, a company should bear in mind business requirements for data, and make their choice based on whether an ETL solution possesses functionalities that meet those requirements, or whether a vendor may add the needed functionality to their solution.

Look. You’ve defined your data integration strategy, business users have created the list of requirements for the data they would need to work with. So now it is clear what data should gather the future ETL tool, and what operations it should perform over that data. Now, having all the necessary criteria, you won’t be wandering blindly among multiple vendors, but will concentrate on those whose ETL solutions meet your criteria.